Neural networks consist of a nested sequence of linear functions followed by pointwise nonlinearities, organized in layers. They have many hyperparamaeters, but when tuned correctly, they scale well with data, so they are typically the method of choice for medium size and big data, trained typically on one or a small number of GPUs. On small data they are usually competitive with kernel methods and forests but need more careful tuning. On image data, convolutional nets are unequivocal state of the art. We list here challenges where neural networks turned to be competitive.
See artificial neural network on Wikipedia.