(Looking for the TLDR troubleshooting list?)
From a discussion on the Darknet/YOLO Discord on 2020-Nov-12:
And the follow-up question:
In older versions of Darknet, OpenCV was optional. In newer forks of Darknet, OpenCV is now required.
See also: Which repos to use and avoid?
The minimum you'll need to download and build is darknet. Darknet is the framework, which also includes the YOLO configuration files. This software will let you both train new neural networks, and run inference using images & video frames.
The original version of Darknet was written by Joseph Redmon, but his "pjreddie" repo was abandoned many years ago and should not be used. It hasn't received any updates since ~2016. This was forked by "AlexeyAB" and maintained from late 2016 to 2021. That was then forked by Hank.ai in 2023. Please see the fork maintained by Stéphane Charette and Hank.ai. This is considered the "current" version of Darknet. The instructions below in the FAQ for building in Windows and building in Linux assumes that this modern version of Darknet is used.
"Darknet" can refer to the overall framework, the CLI tool, or the C library (libdarknet.so in linux or darknet.dll in Windows). The C library allows people to easily incorporate support for neural networks into their own applications.
OpenCV is an open-source library that provides image manipulation functionality. It is a very popular library, and common to many modern projects that work with images. The library is quite complex, has many optional sub-requirements, and many different tutorials exist to do custom/manual builds. Unfortunately, many of these tutorials are for projects other than Darknet, and make incorrect assumptions about what modules from OpenCV are needed.
Older versions of Darknet didn't require OpenCV. As of 2023, the newer versions of Darknet require OpenCV.
CUDA is an optional closed-source library from NVIDIA. It only makes sense to use CUDA if you have a NVIDIA-brand GPU in your computer. For training, this is more-or-less required. For inference, this is optional, though without a CUDA-supported GPU it means inference will be slower.
CUDNN is an optional closed-source library from NVIDIA. It provides functionality that neural networks can use to speed things up. Similarly to CUDA, this requires a NVIDIA-brand GPU in your computer.
DarkHelp is an optional open-source C++ API wrapper and CLI for Darknet. DarkHelp was written by Stéphane Charette -- the author of this FAQ -- to supplement Darknet and provide a robust CLI and commercial-ready API that does more than Darknet itself. DarkHelp provides several advanced features -- such as image tiling, snapping, and DarkHelp Server -- which Darknet does not support natively. DarkHelp is available for Linux, Windows, and probably Mac (not tested).
DarkMark is an optional open-source GUI tool to manage Darknet projects, review annotations, and create Darknet training files. Stéphane Charette provides several dozen youtube videos showing how to use DarkMark to successfully build a Darknet/YOLO neural network. To use DarkHelp's tiling mode to find small objects, DarkMark is typically required as it has the necessary options to pre-process the training images and annotations for tiling. DarkMark requires Linux, though it does also run in Windows when using WSL 2 or in a Linux VM.
See also: Which repos to use and avoid?
Originally, Darknet was written by Joseph Redmon. His web site is still up (as of 2023) and his repo still exists: https://github.com/pjreddie/darknet. The first commit to this repo was on Nov 4, 2013. This is the repo that gave us the original YOLO and YOLOv2 (2016-11).
Note that this repo is out of date, and has not been maintained since late 2016. Do not use this old version of Darknet!
This quickly became the most popular fork of the original Darknet codebase. Also known as the "AlexeyAB" fork, it was most active from late 2016 to July 2021.
This is the repo that gave us YOLOv3 (2018-05), YOLOv4 (2020-05), and YOLOv7 (2022-08). Alexey is now working on other projects and no longer maintains this codebase.
In May 2023, the company Hank.ai announced they will sponsor the continuing development of Darknet/YOLO. The most recent "AlexeyAB" repo was forked again. Stéphane Charette maintains this modern fork of Darknet/YOLO. This is the repo which is currently recommended for use: https://github.com/hank-ai/darknet.
In addition to working on the Hank.ai fork, Stéphane Charette has 2 other repos that may be of interest to Darknet/YOLO users:
The most recent tutorial is the one from December 2023 to find defects in pieces of leather:
As shown in the video, getting the annotations done and training the network for the first time can be accomplished in less than 30 minutes.
In the past, there were 2 other tutorials linked from the Darknet/YOLO FAQ, one for "fire" (February 2021) and the other for "stop signs" (August 2020). Be careful if you view those older tutorials since many of the steps have since changed.
Darknet comes with many configuration files. This is how you select which configuration file to use:
Videos to watch:
The default network sizes in the common template configuration files is defined as 416x416 or 608x608, but those are only examples!
Choose a size that works for you and your images. The only restrictions are:
Whatever size you choose, Darknet will stretch (without preserving the aspect ratio!) your images to be exactly that size prior to processing the image. This includes both training and inference. So use a size that makes sense for you and the images you need to process, but remember that there are important speed and memory limitations:
There are 3 related sizes that must be considered:
Of these 3 sizes, Darknet/YOLO only knows about one: the network dimensions. For example, the default network dimensions (which can be changed!) when using YOLOv4-tiny.cfg is 416x416.
This means that prior to looking at any video frame or image, Darknet will resize the image to match the exact network dimensions in the .cfg file. Darknet/YOLO will never see the original full-size image.
Please re-read the previous paragraph, it is extremely important to understand! It means you can give Darknet/YOLO an image that measures 999999x999999, or an image that measure 500x500, and it will make zero difference. In both cases the images are guaranteed to always be resized to 416x416 prior to Darknet/YOLO examining them.
Now that we know what Darknet/YOLO does with the images, we need to understand how this impacts the results:
Let's jump to an example. Now that we know that all video frames and images will be resized to the network dimensions, it is important to understand what impact that has on the objects within the images. This image of some mailboxes originally measures 1200x900 (click to see the full-sized image):
Here is the same image, but with the training annotations shown:
Remember that Darknet/YOLO only cares about the network dimensions! So assuming we have a network that measures 416x416, here is the same annotated image as above, but resized to match the network dimensions:
The locks which previously seemed to be easy to find now measure only 19x24 pixels:
Anything larger than 16x16 pixels shouldn't be a problem for Darknet/YOLO, but as this shows, it doesn't take long before small objects approach that size once the images are scaled down to match the network dimensions.
(If your objects are smaller than 16x16 pixels once your images are resized to match the network dimensions, see this video on YouTube.)
If instead our images were like this one:
Once the image (with annotations) is resized to match the network dimensions of 416x416, the locks only measure approximately 5x7 pixels. Darknet/YOLO cannot find objects of that size.
If you attempt to train with annotations and images like this with too-small objects, you end up with a network that has a low mAP% and very poor results because during training you're telling Darknet that something exists at this location, but Darknet cannot see it.
As you are reading through the math in this answer, also see this video:
Math can tell you what network dimensions you should be using. Assuming you've already seen the FAQ entries on which configuration file to use, square networks, and that you understand the 3 related sizes, then it should be simple to calculate the best size to use for your network.
Knowing that Darknet/YOLO works best when the object size is 16x16 pixels or larger, take the smallest object you have in your images, and see what size the image needs to be before this object is approximately 16x16 pixels. Any decent image application like GIMP can help. For example, given mailboxes like the ones in this image (click to see the full-size image):
In GIMP, it is easy to see that the smallest objects in the original 1200x900 images are approximately 52x52 pixels in size:
Since the original image size is 1200x900, this means to get an object size that measures approximately 16x16, the image dimensions can be cut by 3. 52x52 / 3 = 17x17. This would make the locks measure 17x17 pixels in the resized images.
And 1200x900 / 3 = 400x300. However, knowing that the network dimensions must be divisible by 32, the closest values to 400x300 would be 416x288. Indeed, when this image is resized to 416x288, the locks are approximately 18x16 pixels which is perfect for Darknet/YOLO:
When you change the size of the network, you'll also want recalculate the optimal YOLO anchors. There are tools like DarkMark which automatically does this, or see the Darknet readme file for details.
See also: How many FPS should I expect to get?
While technically you could, the length of time you'll have to wait does not make sense. Doing inference of single images with a CPU does work (measured in seconds, versus GPU which is measured in milliseconds), but to give you an idea of the difference between CPU and GPU, see the following:
The CPU-only version of darknet can be used for inference, but it is not as fast as when a CUDA-compatible GPU is available.
When using the CPU, inference is measured in seconds. When using a GPU, inference is measured in milliseconds.
An example, using YOLOv4-tiny, 640x480, against a directory with 65 JPG images:
It depends. There isn't a single magic number that answers this question for everyone.
A simple tldr answer: between several hundred and many thousands. If you are asking, it probably means many thousands.
A more detailed answer:
In all cases, you can start with just a few images. In the old stop sign tutorial a neural network is trained to find stop signs with 30+ images. And you can do it with less. But your neural network will be extremely limited. Note that in that tutorial, all signs are more-or-less the same size, taken from similar distances and angles. The network trained will be limited in finding stop signs which are similar to what was used to train.
50% of your dataset should be negative samples. Negative samples are images where your objects do not appear.
For example, if you are detecting people walking in a park, then 50% of your images should be the park without people. This way the network learns the difference between what is important and what is the background to be ignored.
Note that the quality of the negative samples is important. Same example, if you are detecting people walking in a park but your negative samples is pictures of teacups and utensils, it doesn't help the network learn. Instead, your negative samples should be the same -- or similar -- park settings. In summer, winter, night, and day...the same kind of images you'd use to detect people walking in a park.
Also useful would be other things in the park that you don't want to confuse with people. So dogs in the park. Squirrels in the park. Birds in the park. Things which the network needs to learn might appear in your pictures, but are not objects to detect.
Or, to be more precise, you'll probably end up with a neural network that is great at detecting your synthetic images, but unable to detect much in real-world images.
(Yes, one of my first Darknet tutorials was to detect barcodes in synthetic images, and that neural network worked great...but mostly at detecting barcodes in synthetic images!)
This video has additional information on why synthetic images typically don't work well with Darknet/YOLO:
See also: How to annotate images?
If training abruptly stops with the following message:
This means you don't have enough video memory on your GPU card. There are several possible solutions:
If the network size cannot be modified, the most common solution is to increase the subdivision. For example, in the [net] section of your .cfg file you might see this:
Try doubling the subdivisions and train again:
Keep doubling the subdivisions until you don't get the out-of-memory error.
If subdivision=... matches the value in batch=... and you still get an out-of-memory error, then you'll need to decrease the network dimensions (the width=... and height=... in [net]) or select a less demanding configuration.
See also: How much memory does it take?
The Linux instructions for building Darknet are very simple. It should not take more than 2 or 3 minutes to get all the dependencies installed and everything built.
Taken from the Hank.ai Darknet repo page, it should look similar to this when building Darknet on a Debian-based distribution such as Ubuntu:
This YouTube video shows how to install Darknet and associated tools:
If you want to use your NVIDIA GPU -- especially if you plan on training -- see the relevant information in the Hank.ai repo readme.
You can download and install CUDA from here: https://developer.nvidia.com/cuda-downloads.
You can download and install CUDNN from here: https://developer.nvidia.com/rdp/cudnn-download.
See also: What about OpenCV and CUDA?
The Windows builds are slightly more complicated than the Linux ones, but the process is very similar.
Run the following in a command prompt window (not Powershell!):
Add C++ to Visual Studio like this:
Once this is done, install OpenCV on Windows. Open a Developer Command Prompt for VS 2022 window and run the following commands:
Building OpenCV and dependencies will take some time. Once everything is built, run the following commands:
Additional details -- including installation instructions for CUDA and CUDNN -- are available in the Hank.ai repo readme. Remember that on Windows, CUDA and CUDNN must be installed after Visual Studio. If you upgrade your installation of Visual Studio, you also must remember to re-install CUDA and CUDNN.
You definitely should NOT build OpenCV by hand! Yes, we know about the many tutorials. But it is difficult to do it correctly, and very easy to get it wrong. OpenCV is a complex library, with many optional modules. Some of those modules are not optional for Darknet, and many OpenCV tutorials don't include them.
Instead, please follow the standard way to install OpenCV for your operating system. On Debian-based Linux distributions, it is a single command that should look similar to this:
On Windows, use Microsoft's vcpkg to install OpenCV correctly:
Note there are other modules which may be necessary. See Darknet's build.ps1 (from a previous version, now out of date) for examples of what may be needed.
That's it! It really should never be more complicated than that. On Linux it shouldn't take more than a few seconds, while on Windows it will take some time to build.
Which brings up the topic of CUDA-enabled OpenCV. Please take note: Darknet DOES NOT use the CUDA portion of OpenCV! Don't bother spending hours (days?) trying to install it, Darknet wont use it. At all!
Darknet does use CUDA to train and run the neural network, but it does so directly, not via OpenCV. OpenCV is used to load images from disk, resize images, and data augmentation such as the "mosaic" images, all of which is done without the GPU.
To use CUDA and the GPU from within OpenCV, you'd need to make extensive modifications to the Darknet source code. For example, to use cv::cuda::GpuMat instead of the usual cv::Mat. And even then, it isn't as simple as people think, as described in this blog post on CUDA and OpenCV.
So save yourself the headache, install the "normal" OpenCV libraries, and don't waste any more time on trying to figure out how to enable CUDA within OpenCV.
Advanced topic (aka "can of worms"):
If you are using OpenCV's DNN module + CUDA to do your inference instead of Darknet, then obviously you need CUDA support when building OpenCV. This is an advanced topic, and isn't how most people starting out with Darknet/YOLO run their neural networks.
But this only applies to advanced users once they have everything already running.
See also: How many FPS should I expect to get?
This depends on several things:
Please see this blog post with details on running Darknet/YOLO/OpenCV on several popular IoT devices, like the Raspberry Pi, Jetson Nano, Jetson NX, etc.
There are several factors that determine how much video memory is needed on your GPU to train a network. Except for the first (the configuration file itself), these are all defined in the [net] section at the top of the configuration file:
Typically, once a network configuration and dimensions are chosen, the value that gets modified to make the network fit in the available memory is the batch subdivision.
You'll want the subdivision to be as small as possible without causing an out-of-memory error. Here are some values showing the amount of GPU memory required using various configurations and subdivisions:
Here is the same table but for a slightly larger network size:
Memory values as reported by nvidia-smi. My setup is a GeForce RTX 2070 with only 8 GiB of memory, which limits the configurations I can run.
The length of time it takes to train a network depends on the input image data, the network configuration, the available hardware, how Darknet was compiled, even the format of the images at extremes.
Some tldr notes:
The format of the images -- JPG or PNG -- has no meaningful impact on the length of time it takes to train unless the images are excessively large. When very large photo-realistic image files are saved as PNG, the excessive file sizes means loading the images from disk is slow, which significantly impacts the training time. This should never be an issue when the image sizes match the network sizes.
The table shows the length of time it takes to train a neural network:
|original 4608x3456 JPG images
|4608x3456 JPG images, quality=75
|800x600 JPG images, quality=75
|416x416 JPG images, quality=75
|Darknet compiled to use GPU + OpenCV
|10 iterations: 42.26 seconds
10K iterations: 11h 44m
|10 iterations: 35.27 seconds
10K iterations: 9h 47m
|10 iterations: 6.90 seconds
10K iterations: 1h 55m
|10 iterations: 6.76 seconds
10K iterations: 1h 53m
|Darknet compiled to use GPU + OpenCV,
but using PNG images instead of JPG
|10 iterations: 80.70 seconds
10K iterations: 22h 25m
|10 iterations: 6.93 seconds
10K iterations: 1h 56m
|10 iterations: 6.71 seconds
10K iterations: 1h 52m
|Darknet compiled to use GPU, but without OpenCV
|10 iterations: 113.31 seconds
10K iterations: 31h 29m
|10 iterations: 106.56 seconds
10K iterations: 29h 36m
|10 iterations: 9.19 seconds
10K iterations: 2h 33m
|10 iterations: 7.70 seconds
10K iterations: 2h 8m
|Darknet compiled for CPU + OpenCV (no GPU)
|10 iterations: 532.86 seconds
10K iterations: > 6 days
|10 iterations: 527.41 seconds
10K iterations: > 6 days
|10 iterations: 496.47 seconds
10K iterations: > 5 days
|10 iterations: 496.03 seconds
10K iterations: > 5 days
For these tests, GPU was a GeForce RTX 2070 with 8 GiB of memory, CPU was a 8-core 3.40 GHz.
Note that all the neural networks trained in the previous table are exactly the same. The training images are identical, the validation images are the same, and the resulting neural networks are virtually identical. But the length of time it takes to train varies between ~2 hours and 6+ days.
See also: What is the optimal network size.
I store all my neural network projects in subfolders within /home/username/nn/ (where "nn" means "neural networks"). So if I was to start a project called "animals", all my images and configuration files would be stored in ~/nn/animals/.
This is the command I would run to train my "animals" neural network from within Linux:
Note how I don't use any starting weights files. When I train my own neural networks, I always start with a clean slate.
I like to capture all the output generated by darknet (STDERR and STDOUT) so modify the previous command like this:
On Windows, if you don't have a tool similar to tee, then the command might look like this:
Darknet does not need to run from within the darknet subdirectory. It is a self-contained application. You can run it from anywhere as long as it is on the path, or you specify it by name.
The various filenames (data, cfg, images, ...) can be relative to the current directory, but I prefer to use absolute filenames in all the darknet configuration files to remove any ambiguity. For example, given the previous "animals" project, the content of the animals.data file might look like this:
Once training has started, open the image file chart.png in your favourite image viewer to see the progress. Darknet regularly updates chart.png during training.
If you want to re-train your network using previous weights, you'll need to add the -clear flag, as well as the name of the existing weights to use. So to continue with the "animals" example, that would look like this:
Important flags to know about when training:
The Darknet repo is cloned into the ~/src/ subdirectory. Don't put any other personal or modified files into that subdirectory to maintain a clean source directory. This way it will be very simple to "git pull" every once in a while to grab the most recent Darknet changes.
The directory should look like this:
All of the trained neural networks are stored in a completely different subdirectory called "nn". Each project is a different subdirectory from within "nn".
For example, in this image I have 4 custom neural network projects:
Let's use the "fire" project as an example.
Within the /home/stephane/nn/fire/ subdirectory, I create one or more image dataset directory, and I also create the .names file.
My "fire" project then looks like this:
Lastly, the rest of the darknet configuration files are also written out to this project directory. (Not the Darknet source code directory!) This includes the .cfg, the .data, and the various other files created by darknet and DarkMark.
Note how I use absolute paths in the .data and various .txt files to remove any ambiguity as to where the files reside.
Once everything is setup this way, I call darknet directly to begin training, and I continue using absolute path names. For example:
On Windows the instructions above remain more-or-less the same but replace all instances of /home/stephane/nn/fire/ with C:/Users/stephane/Desktop/nn/fire/.
Contrary to what most people would initially guess, the valid=... line in the .data file is not used for training the neural network! The file is only used to draw the results in chart.png. So especially if you have very few images in your dataset, go ahead and use 100% of the the annotated images for training, and the same 100% for validation.
The conversation with AlexeyAB on this topic was as follows:
The answer from AlexeyAB was:
The preferred way would be to use the API. This way you load the network once, run it against as many images you need, and process the results exactly the way you want.
Darknet has a C API, C++ bindings, and there are other open-source libraries such as DarkHelp which provide an alternate C++ API.
To use the darknet command line instead of the API, search for the text "list of images" in the Darknet readme. It gives a few examples showing how to process many images at once to get the results in JSON format.
Similarly, DarkHelp also has a JSON/CLI mode which may be used to process many images at once.
Say you want a network trained to find barcodes. If you crop and label your training images like this:
...then your network will only learn to recognize barcodes when they take up 100% of the image. It is unlikely you want that; if the objects to find always took up 100% of the image, then there is little use to train a neural network.
Instead, make sure your training images are representative of the final images. Using this barcode as an example, a more likely marked up training image would be similar to this:
See also: Darknet & DarkMark image markup.
Example video that shows how to use DarkMark to annotate images for use with Darknet/YOLO:
Images that don't contain any of the objects you want to find are called negative samples, and they are important to have in your training set. When marking up your images, the negative samples will have a blank (empty) .txt file, telling Darknet that nothing of interest exists in that image.
In DarkMark, this is done by selecting the "empty image" annotation.
Meanwhile, the rest of your images should have 1 annotation per object. If an image contains 3 vehicles, and "vehicle" is one of your classes, then you must markup the image 3 times, once for each vehicle. Don't use a single large annotation that covers all 3 vehicles, unless that is what you want the network to learn. And similarly, don't break up your object into multiple smaller parts to try and get better or tighter coverage. You should stick to the rule "1 object = 1 annotation".
This becomes much more challenging when trying to detect things like clouds, smoke, fire, or rain: the goal is not to cover the image with many small annotations to achieve 100% pixel coverage. Instead, you want to identify each distinct object you'd like the neural network to identify.
Image annotated incorrectly:
Same image annotated correctly:
Additional markup comments and techniques are discussed on DarkMark's "markup" page.
See also: How many images?
Darknet/YOLO annotations are saved to text files, one per image. Each line is a bounding box within the image. The annotation coordinates are normalized, and look like this:
The five fields are:
Here is an example to demonstrate. Say we want to annotate the "5" mailbox digit in this image:
Knowing the center of the "5" is located at coordinate (240, 99.5) and the image dimensions are 560x420, the values are normalized like this:
The classes for this neural network are the digits from "0" to "9", so the first value at the start of the line would be "5". The annotation in the .txt file would look like this:
If you are using DarkMark, then set to zero or turn off all data augmentation options.
If you are editing your configuration file by hand, verify these settings in the [net] section:
Depends on the type of image. Some things don't make much sense rotated (e.g., dashcam or highway cam images). But the impact of rotated images needs to be considered. For example, here is a network that is really good at detecting animals:
With 100% certainty, that is a very cute dog. But when the exact same image is rotated 180 degrees, all of a sudden the neural network thinks this is more likely to be a cat than a dog.
See also: Data Augmentation - Rotation
Help support the development and maintenance of Darknet/YOLO, DarkHelp, and DarkMark with the purchase of items from the store.
Come see us on the Darknet/YOLO Discord!