../
(Looking for the TLDR troubleshooting list?)

Summary

The Darknet/YOLO Discord and Darknet Issues regularly sees certain questions come up. This page will attempt to answer these frequenty asked questions.

🔗

Stéphane's Darknet/YOLO FAQ:

🔗

What is Darknet? What is YOLO?

From a discussion on the Darknet/YOLO Discord on 2020-Nov-12:

> What is a "Darknet" and what is a "Yolo"? Darknet is a tool -- or framework -- written mostly in C which may be used for computer vision. It is used to both train neural networks, and then run images or video frames through those neural networks. YOLO is one of many possible configurations files that Darknet uses. These text configuration files define how neural networks are built and behave. Some config files describe networks which are fast but not precise. Others are very precise, but take relatively long time to run. And plenty are somewhere in the middle. Every configuration file has different pros and cons. YOLOv4 is the most recent version of YOLO. YOLOv4-tiny is a smaller/faster version of YOLOv4. And YOLOv4-tiny-3l is a 3-layer version which is somewhere between YOLOv4 and YOLOv4-tiny.

And the follow-up question:

> I dont understand what is neural network from this stack: Darknet, Yolov, OpenCV. - Darknet is a tool that trains and uses neural networks. - A neural network is software that can be used to find patterns in things. In the case of darknet, the neural networks it creates -- which Darknet calls "weights" files -- are used to find patterns in images. Different tools can create neural networks that can find patterns in sound, financial data, numerical data, etc. You have to "train" a neural network to show it which patterns ("objects") you want to find. - OpenCV is a software library originally by Intel that is often used by programmers working with images. It lets you do generic things with images, including finding edges so you can use it to perform some simple object detection. Over the last 10+ years, neural networks have proven to be generally better and faster than using OpenCV directly. OpenCV now has support for running (not training) existing neural networks, so you can take your Darknet neural network and load it directly into OpenCV.

Darknet does not require OpenCV and may be built without it. But when OpenCV is available both training and inference is faster.

See also: Which repos to use and avoid?

🔗

What software is used to make this work?

Darknet

The minimum you'll need to download and build is darknet. Darknet is the framework, which also includes the YOLO configuration files. This software will let you both train new neural networks, and run inference using images & video frames.

The original version of Darknet was written by Joseph Redmon, but his "pjreddie" repo was abandoned many years ago and should not be used. It hasn't received any updates since ~2016. This was forked by "AlexeyAB" and maintained from late 2016 to 2021. That was then forked by Hank.ai in 2023. Please see the fork maintained by Stéphane Charette and Hank.ai. This is considered the "current" version of Darknet. The instructions below in the FAQ for building in Windows and building in Linux assumes that this modern version of Darknet is used.

"Darknet" can refer to the overall framework, the CLI tool, or the C library (libdarknet.so in linux or darknet.dll in Windows). The C library allows people to easily incorporate support for neural networks into their own applications.

OpenCV

OpenCV is an optional open-source library that provides image manipulation functionality. It is a very popular library, and common to many modern projects that work with images. The library is quite complex, has many other optional sub-requirements, and many different tutorials exist to do custom/manual builds. Unfortunately, many of these tutorials are for projects other than Darknet, and make incorrect assumptions about what modules from OpenCV are needed.

Darknet may be built without the use of OpenCV, but this limits what Darknet can do and also slows down training.

CUDA

CUDA is an optional closed-source library from NVIDIA. It only makes sense to use CUDA if you have a NVIDIA-brand GPU in your computer. For training, this is more-or-less required. For inference, this is optional, though without a CUDA-supported GPU it means inference will be slower.

CUDNN

CUDNN is an optional closed-source library from NVIDIA. It provides functionality that neural networks can use to speed things up. Similarly to CUDA, this requires a NVIDIA-brand GPU in your computer.

DarkHelp

DarkHelp is an optional open-source C++ API wrapper and CLI for Darknet. DarkHelp was written by Stéphane Charette -- the author of this FAQ -- to supplement Darknet and provide a robust CLI and commercial-ready API that does more than Darknet itself. DarkHelp provides several advanced features -- such as image tiling, snapping, and DarkHelp Server -- which Darknet does not support natively. DarkHelp is available for Linux, Windows, and probably Mac (not tested).

DarkMark

DarkMark is an optional open-source GUI tool to manage Darknet projects, review annotations, and create Darknet training files. Stéphane Charette provides several dozen youtube videos showing how to use DarkMark to successfully build a Darknet/YOLO neural network. To use DarkHelp's tiling mode to find small objects, DarkMark is typically required as it has the necessary options to pre-process the training images and annotations for tiling. DarkMark requires Linux, though it does also run in Windows when using WSL 2 or in a Linux VM.

See also: Which repos to use and avoid?

🔗

Which software repos should I use? Which should I avoid?

Joseph Redmon

Originally, Darknet was written by Joseph Redmon. His web site is still up (as of 2023) and his repo still exists: https://github.com/pjreddie/darknet. The first commit to this repo was on Nov 4, 2013. This is the repo that gave us the original YOLO.

Note that this repo is out of date, and has not been maintained since late 2016. Do not use this old version of Darknet!

Alexey Bochkovskiy

This quickly became the most popular fork of the original Darknet codebase. Also known as the "AlexeyAB" fork, it was most active from late 2016 to July 2021.

This is the repo that gave us YOLOv2 (2018-03), YOLOv3 (2018-05), YOLOv4 (2020-05), and YOLOv7 (2022-08). Alexey is now working on other projects and no longer maintains this codebase.

Hank.ai

In May 2023, the company Hank.ai announced they will sponsor the continuing development of Darknet/YOLO. The most recent "AlexeyAB" repo was forked again. Stéphane Charette maintains this modern fork of Darknet/YOLO. This is the repo which is currently recommended for use: https://github.com/hank-ai/darknet.

Stéphane Charette

In addition to working on the Hank.ai fork, Stéphane Charette has 2 other repos that may be of interest to Darknet/YOLO users:

🔗

How to get started with Darknet? How long will it take?

The simple stop sign tutorial (also available as a YouTube video) is a good place to get started. It will probably take you ~90 minutes to complete:

At the end, you'll have a simple custom neural network that can find stop signs.

There is a 2nd different "getting started" tutorial in the section that discusses How to build Darknet on Linux?.

See also: How many images will it take? and How long does it take to train?

🔗

Which configuration file should I use?

Darknet comes with many configuration files. This is how you select which configuration file to use:

Videos to watch:

See also: Configuration Files and How many FPS should I expect to get?

🔗

Does the network have to be perfectly square?

No.

The default network sizes in the common template configuration files is defined as 416x416 or 608x608, but those are only examples!

Choose a size that works for you and your images. The only restrictions are:

Whatever size you choose, Darknet will stretch (without preserving the aspect ratio!) your images to be exactly that size prior to processing the image. This includes both training and inference. So use a size that makes sense for you and the images you need to process, but remember that there are important speed and memory limitations:

See also: How does sizing work, What is the optimal network size, and How much memory does it take?

🔗

How does sizing work?

This is somewhat already discussed on the DarkHelp and DarkMark pages. But here is a simple example to visually understand what is happening.

There are 3 related sizes that must be considered:

  1. the image dimensions
  2. the network dimensions
  3. the size of each object within the images

Of these 3 sizes, Darknet/YOLO only knows about one: the network dimensions. For example, the default network dimensions (which can be changed!) when using YOLOv4-tiny.cfg is 416x416.

This means that prior to looking at any video frame or image, Darknet will resize the image to match the exact network dimensions in the .cfg file. Darknet/YOLO will never see the original full-size image.

Please re-read the previous paragraph, it is extremely important to understand! It means you can give Darknet/YOLO an image that measures 999999x999999, or an image that measure 500x500, and it will make zero difference. In both cases the images are guaranteed to always be resized to 416x416 prior to Darknet/YOLO examining them.

Now that we know what Darknet/YOLO does with the images, we need to understand how this impacts the results:

Let's jump to an example. Now that we know that all video frames and images will be resized to the network dimensions, it is important to understand what impact that has on the objects within the images. This image of some mailboxes originally measures 1200x900 (click to see the full-sized image):

image of a Canadian community mailbox

Here is the same image, but with the training annotations shown:

annotated image of a Canadian community mailbox

Remember that Darknet/YOLO only cares about the network dimensions! So assuming we have a network that measures 416x416, here is the same annotated image as above, but resized to match the network dimensions:

annotated image of a Canadian community mailbox

The locks which previously seemed to be easy to find now measure only 19x24 pixels:

zoom

Anything larger than 16x16 pixels shouldn't be a problem for Darknet/YOLO, but as this shows, it doesn't take long before small objects approach that size once the images are scaled down to match the network dimensions.

(If your objects are smaller than 16x16 pixels once your images are resized to match the network dimensions, see this video on YouTube.)

If instead our images were like this one:

more mailboxes

Once the image (with annotations) is resized to match the network dimensions of 416x416, the locks only measure approximately 5x7 pixels. Darknet/YOLO cannot find objects of that size.

zoom

If you attempt to train with annotations and images like this with too-small objects, you end up with a network that has a low mAP% and very poor results because during training you're telling Darknet that something exists at this location, but Darknet cannot see it.

See also: Does the network have to be perfectly square and What is the optimal network size.

🔗

What is the optimal network size?

As you are reading through the math in this answer, also see this video:

Math can tell you what network dimensions you should be using. Assuming you've already seen the FAQ entries on which configuration file to use, square networks, and that you understand the 3 related sizes, then it should be simple to calculate the best size to use for your network.

Knowing that Darknet/YOLO works best when the object size is 16x16 pixels or larger, take the smallest object you have in your images, and see what size the image needs to be before this object is approximately 16x16 pixels. Any decent image application like GIMP can help. For example, given mailboxes like the ones in this image (click to see the full-size image):

annotated image of a Canadian community mailbox

In GIMP, it is easy to see that the smallest objects in the original 1200x900 images are approximately 52x52 pixels in size:

zoom

Since the original image size is 1200x900, this means to get an object size that measures approximately 16x16, the image dimensions can be cut by 3. 52x52 / 3 = 17x17. This would make the locks measure 17x17 pixels in the resized images.

And 1200x900 / 3 = 400x300. However, knowing that the network dimensions must be divisible by 32, the closest values to 400x300 would be 416x288. Indeed, when this image is resized to 416x288, the locks are approximately 18x16 pixels which is perfect for Darknet/YOLO:

zoom

When you change the size of the network, you'll also want recalculate the optimal YOLO anchors. There are tools like DarkMark which automatically does this, or see the Darknet readme file for details.

See also: How many FPS should I expect to get?

🔗

Can I train a neural network with a CPU?

Not really.

While technically you could, the length of time you'll have to wait does not make sense. Doing inference of single images with a CPU does work (measured in seconds, versus GPU which is measured in milliseconds), but to give you an idea of the difference between CPU and GPU, see the following:

See also: How long does it take and CPU vs GPU.

🔗

What is the difference between CPU and GPU performance?

The CPU-only version of darknet can be used for inference, but it is not as fast as when a CUDA-compatible GPU is available.

When using the CPU, inference is measured in seconds. When using a GPU, inference is measured in milliseconds.

An example, using YOLOv4-tiny, 640x480, against a directory with 65 JPG images:

See also: Training with a CPU? and How many FPS should I expect to get?

🔗

How many images will it take?

It depends. There isn't a single magic number that answers this question for everyone.

A simple tldr answer: between several hundred and many thousands. If you are asking, it probably means many thousands.

A more detailed answer:

  1. Say your application is a camera pointed to a fixed location inside a machine, or looking at a spot on a conveyer belt in a factory. We'll call this scenario #1.
  2. Then we can imagine a camera pointed to a fixed location outdoors. Such as a traffic cam mounted to a pole and always looking at the exact same scene. We'll call this scenario #2.
  3. The last example will be for drones that fly around outdoors, or for people to use random images they either download on the internet or take by themselves, where there is no control of the size of the image, the zoom, the angle, the quality of the image, etc. This is scenario #3.

In all cases, you can start with just a few images. In the stop sign tutorial a neural network is trained to find stop signs with 30+ images. And you can do it with less. But your neural network will be extremely limited. Note that in that tutorial, all signs are more-or-less the same size, taken from similar distances and angles. The network trained will be limited in finding stop signs which are similar to what was used to train.

  1. Scenario #1: In the first scenario, there is an extremely low amount of variability in what the camera will see. You can train with very few images. It is very important in these scenarios to have negative samples during training. So if the network needs to find a part moving on a conveyer belt, then make sure to include images the camera will see of the conveyer belt without the part you need it to find. (This is mentioned briefly in the Darknet readme, search for "negative samples".) To get started, annotate 2 dozen images and include an additional dozen negative sample images. (See also: What about negative samples? and How to annotate images?) Depending on variability with ligthing conditions, object position, and how many ways the part can be rotated, you'll eventually work up to hundreds of images to get a decent network trained.
  2. Scenario #2: In the second scenario, the scene might change in lighting, but the background is relatively fixed. Like the previous scenario, negative samples will help the neural network determine what is the background that must be ignored. You may need to collect images over the span of a year to eventually train the network the difference between summer, winter, and everything in between. To get started you'll have a few dozen images, but know you need to immediately scale to thousands. And the more classes you want to detect, the more images you need. For example:
    • Need to recognize a blue car? That is an image.
    • A blue car in the lower right as well as the upper left? More images.
    • Do you need to find red cars too? Duplicate all the previous images.
    • Cars of any colour of the rainbow? Yet more images.
    • Small cars as well as large cars?
    • What about trucks? Pickup truck, transport trucks, garbage truck? More images of each type and colour.
    • Buses? School buses? City buses? Commercial buses? Black buses? White buses? More images!
    • Cars at noon? Cars when it is raining? Cars in winter when the road is white and covered in snow? At 9am or 9pm? With headlights on or with lights off?
    Each of these scenarios barely begins to describe all the possibilities. And if you want the neural network to function correctly, you need annotated images of each possibility.
  3. Scenario #3: The last scenario is the most complex. Because you have all the possibilities of the last scenario, but in addition you don't control who takes the images, the type of camera used, or how it is done. You have no idea what the background will be. You have no idea if the camera will be in focus, zoomed in, tilted slightly to one side, what the lighting conditions are, if the image is upside down, etc. So of course to get started you can begin with just a few dozen images, but because diversity is key to make this scenario work, your network will need thousands of annotated images per class.

See also: What about negative samples? and How to annotate images?

🔗

What about negative samples?

50% of your dataset should be negative samples. Negative samples are images where your objects do not appear.

For example, if you are detecting people walking in a park, then 50% of your images should be the park without people. This way the network learns the difference between what is important and what is the background to be ignored.

Note that the quality of the negative samples is important. Same example, if you are detecting people walking in a park but your negative samples is pictures of teacups and utensils, it doesn't help the network learn. Instead, your negative samples should be the same -- or similar -- park settings. In summer, winter, night, and day...the same kind of images you'd use to detect people walking in a park.

Also useful would be other things in the park that you don't want to confuse with people. So dogs in the park. Squirrels in the park. Birds in the park. Things which the network needs to learn might appear in your pictures, but are not objects to detect.

See also: How many images will it take? and How to annotate images?

🔗

Can I train a neural network using synthetic images?

No.

Or, to be more precise, you'll probably end up with a neural network that is great at detecting your synthetic images, but unable to detect much in real-world images.

(Yes, one of my first Darknet tutorials was to detect barcodes in synthetic images, and that neural network worked great...but mostly at detecting barcodes in synthetic images!)

This video has additional information on why synthetic images typically don't work well with Darknet/YOLO:

See also: How to annotate images?

🔗

What does the error message "CUDA Error: out of memory" mean during training?

If training abruptly stops with the following message:

Try to set subdivisions=64 in your cfg-file. CUDA Error: out of memory: File exists darknet: ./src/utils.c:325: error: Assertion `0' failed. Command terminated by signal 6

This means you don't have enough video memory on your GPU card. There are several possible solutions:

  1. choose a different configuration file that consumes less memory
  2. decrease the network size, meaning width=... and height=... in the [net] section of your .cfg file
  3. increase the subdivisions, meaning subdivisions=... in the [net] section of your .cfg file
  4. purchase a different GPU with more memory

If the network size cannot be modified, the most common solution is to increase the subdivision. For example, in the [net] section of your .cfg file you might see this:

batch=64 subdivisions=2

Try doubling the subdivisions and train again:

batch=64 subdivisions=4

Or:

batch=64 subdivisions=8

Keep doubling the subdivisions until you don't get the out-of-memory error.

If subdivision=... matches the value in batch=... and you still get an out-of-memory error, then you'll need to decrease the network dimensions (the width=... and height=... in [net]) or select a less demanding configuration.

See also: How much memory does it take?

🔗

How to build Darknet on Linux?

The Linux instructions for building Darknet are very simple. It should not take more than 2 or 3 minutes to get all the dependencies installed and everything built.

Taken from the DarkHelp page, it should look similar to this when building Darknet on a Debian-based distribution such as Ubuntu:

sudo apt-get install build-essential git libopencv-dev mkdir ~/src cd ~/src git clone https://github.com/hank-ai/darknet.git cd darknet # edit Makefile to set LIBSO=1, and possibly other flags make sudo cp libdarknet.so /usr/local/lib/ sudo cp include/darknet.h /usr/local/include/ sudo ldconfig

When you edit the Makefile for the first time, if you're unsure of which flags to set, try this to get started:

GPU=0 CUDNN=0 CUDNN_HALF=0 OPENCV=1 AVX=1 OPENMP=1 LIBSO=1 ZED_CAMERA=0

These are safe options that will build a CPU-only version of Darknet, which doesn't require any NVIDIA files to be installed. Remember this is only to help you get started.

This YouTube video shows how to install Darknet as described above. Watch the "Darknet" segment that begins at 0:50:

If you want to use your GPU -- especially if you plan on training -- go back and enable CUDA and CUDNN after you confirm everything builds and is working.

You can download and install CUDA from here: https://developer.nvidia.com/cuda-downloads.

You can download and install CUDNN from here: https://developer.nvidia.com/rdp/cudnn-download.

See also: What about OpenCV and CUDA?

🔗

How to build Darknet on Windows?

The Windows builds are more complicated and tend to be more fragile than the Linux ones, but with the many build changes merged in early 2021, the process should be much simpler than it was in the past.

Run the following commands in a Microsoft Powershell window:

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process md c:\src cd c:\src git clone https://github.com/hank-ai/darknet.git cd darknet .\build.ps1

The other option is to use Microsoft's vcpkg directly, similar to this:

md c:\src cd c:\src git clone https://github.com/microsoft/vcpkg cd vcpkg bootstrap-vcpkg.bat vcpkg.exe integrate install vcpkg.exe integrate powershell vcpkg.exe install darknet[opencv-base]:x64-windows

If you plan on using your GPU, then you'll need to install CUDA and CUDNN first.

You can download and install CUDA from here: https://developer.nvidia.com/cuda-downloads.

You can download and install CUDNN from here: https://developer.nvidia.com/rdp/cudnn-download.

🔗

What about OpenCV and CUDA?

You definitely should NOT build OpenCV by hand! Yes, we know about the many tutorials. But it is difficult to do it correctly, and very easy to get it wrong. OpenCV is a complex library, with many optional modules. Some of those modules are not optional for Darknet, and many OpenCV tutorials don't include them.

Instead, please follow the standard way to install OpenCV for your operating system. On Debian-based Linux distributions, it is a single command that should look similar to this:

sudo apt-get install libopencv-dev

On Windows, use Microsoft's vcpkg to install OpenCV correctly:

mkdir c:\src cd c:\src git clone https://github.com/microsoft/vcpkg cd vcpkg bootstrap-vcpkg.bat vcpkg.exe integrate install vcpkg.exe integrate powershell vcpkg.exe install opencv[contrib,core,dnn,ffmpeg,jpeg,png,quirc,tiff,webp]:x64-windows

Note there are other modules which may be necessary. See Darknet's build.ps1 for examples of what may be needed.

That's it! It really should never be more complicated than that. On Linux it shouldn't take more than a few seconds, while on Windows it will take some time to build.

Which brings up the topic of CUDA-enabled OpenCV. Please take note: Darknet DOES NOT use the CUDA portion of OpenCV! Don't bother spending hours (days?) trying to install it, Darknet wont use it. At all!

Darknet does use CUDA to train and run the neural network, but it does so directly, not via OpenCV. OpenCV is used to load images from disk, resize images, and data augmentation such as the "mosaic" images, all of which is done without the GPU.

To use CUDA and the GPU from within OpenCV, you'd need to make extensive modifications to the Darknet source code. For example, to use cv::cuda::GpuMat instead of the usual cv::Mat. And even then, it isn't as simple as people think, as described in this blog post on CUDA and OpenCV.

So save yourself the headache, install the "normal" OpenCV libraries, and don't waste any more time on trying to figure out how to enable CUDA within OpenCV.

Advanced topic (aka "can of worms"):

If you are using OpenCV's DNN module + CUDA to do your inference instead of Darknet, then obviously you need CUDA support when building OpenCV. This is an advanced topic, and isn't how most people starting out with Darknet/YOLO run their neural networks.

But this only applies to advanced users once they have everything already running.

See also: How many FPS should I expect to get?

🔗

How many FPS should I expect to get? How can I increase my FPS?

This depends on several things:

Please see this blog post with details on running Darknet/YOLO/OpenCV on several popular IoT devices, like the Raspberry Pi, Jetson Nano, Jetson NX, etc.

🔗

How much memory does it take to train a custom network?

There are several factors that determine how much video memory is needed on your GPU to train a network. Except for the first (the configuration file itself), these are all defined in the [net] section at the top of the configuration file:

Typically, once a network configuration and dimensions are chosen, the value that gets modified to make the network fit in the available memory is the batch subdivision.

You'll want the subdivision to be as small as possible without causing an out-of-memory error. Here are some values showing the amount of GPU memory required using various configurations and subdivisions:

subdivisions= 64 32 16 8 4 2 1
YOLOv3 3085 MiB 4406 MiB 6746 MiB ? ? ? ?
YOLOv3-tiny 1190 MiB 1204 MiB 1652 MiB 2522 MiB 4288 MiB ? ?
YOLOv3-tiny-3l 1046 MiB 1284 MiB 1814 MiB 2810 MiB 4846 MiB ? ?
YOLOv4 4236 MiB 6246 MiB ? ? ? ? ?
YOLOv4-tiny 814 MiB 956 MiB 1321 MiB 1752 MiB 2770 MiB 5532 MiB ?
YOLOv4-tiny-3l 830 MiB 1085 MiB 1282 MiB 1862 MiB 2982 MiB 5748 MiB ?

Here is the same table but for a slightly larger network size:

subdivisions= 64 32 16 8 4 2 1
YOLOv3 4648 MiB 4745 MiB ? ? ? ? ?
YOLOv3-tiny 1278 MiB 1774 MiB 2728 MiB 4634 MiB ? ? ?
YOLOv3-tiny-3l 1473 MiB 2059 MiB 3044 MiB 5420 MiB ? ? ?
YOLOv4 6906 MiB ? ? ? ? ? ?
YOLOv4-tiny 984 MiB 1262 MiB 1909 MiB 2902 MiB 5076 MiB ? ?
YOLOv4-tiny-3l 1020 MiB 1332 MiB 1938 MiB 3134 MiB 5518 MiB ? ?

Memory values as reported by nvidia-smi. My setup is a GeForce RTX 2070 with only 8 GiB of memory, which limits the configurations I can run.

🔗

How long does it take to train a custom network?

The length of time it takes to train a network depends on the input image data, the network configuration, the available hardware, how Darknet was compiled, even the format of the images at extremes.

Some tldr notes:

  1. Resize your training and validation images to match exactly the network size.
    • For example: mogrify -verbose -strip -resize 416x416! -quality 75 *.JPG
  2. Build Darknet with support for OpenCV: loading images is slower without OpenCV.
  3. Build Darknet with support for OpenCV: resizing images is slower without OpenCV.
  4. Build Darknet with support for CUDA/CUDNN. (This requires supported hardware.)
  5. Use the tiny variants of the network.

The format of the images -- JPG or PNG -- has no meaningful impact on the length of time it takes to train unless the images are excessively large. When very large photo-realistic image files are saved as PNG, the excessive file sizes means loading the images from disk is slow, which significantly impacts the training time. This should never be an issue when the image sizes match the network sizes.

The table shows the length of time it takes to train a neural network:

  original 4608x3456 JPG images 4608x3456 JPG images, quality=75 800x600 JPG images, quality=75 416x416 JPG images, quality=75
Darknet compiled to use GPU + OpenCV 10 iterations: 42.26 seconds
10K iterations: 11h 44m		 10 iterations: 35.27 seconds
10K iterations: 9h 47m		 10 iterations: 6.90 seconds
10K iterations: 1h 55m		 10 iterations: 6.76 seconds
10K iterations: 1h 53m
Darknet compiled to use GPU + OpenCV,
but using PNG images instead of JPG		 n/a 10 iterations: 80.70 seconds
10K iterations: 22h 25m		 10 iterations: 6.93 seconds
10K iterations: 1h 56m		 10 iterations: 6.71 seconds
10K iterations: 1h 52m
Darknet compiled to use GPU, but without OpenCV 10 iterations: 113.31 seconds
10K iterations: 31h 29m		 10 iterations: 106.56 seconds
10K iterations: 29h 36m		 10 iterations: 9.19 seconds
10K iterations: 2h 33m		 10 iterations: 7.70 seconds
10K iterations: 2h 8m
Darknet compiled for CPU + OpenCV (no GPU) 10 iterations: 532.86 seconds
10K iterations: > 6 days		 10 iterations: 527.41 seconds
10K iterations: > 6 days		 10 iterations: 496.47 seconds
10K iterations: > 5 days		 10 iterations: 496.03 seconds
10K iterations: > 5 days

For these tests, GPU was a GeForce RTX 2070 with 8 GiB of memory, CPU was a 8-core 3.40 GHz.

Note that all the neural networks trained in the previous table are exactly the same. The training images are identical, the validation images are the same, and the resulting neural networks are virtually identical. But the length of time it takes to train varies between ~2 hours and 6+ days.

See also: What is the optimal network size.

🔗

What command should I use when training my own network?

I store all my neural network projects in subfolders within /home/username/nn/ (where "nn" means "neural networks"). So if I was to start a project called "animals", all my images and configuration files would be stored in ~/nn/animals/.

This is the command I would run to train my "animals" neural network from within Linux:

cd ~/nn/animals/ ~/src/darknet/darknet detector -map -dont_show train ~/nn/animals/animals.data ~/nn/animals/animals.cfg

Note how I don't use any starting weights files. When I train my own neural networks, I always start with a clean slate.

I like to capture all the output generated by darknet (STDERR and STDOUT) so modify the previous command like this:

cd ~/nn/animals/ ~/src/darknet/darknet detector -map -dont_show train ~/nn/animals/animals.data ~/nn/animals/animals.cfg 2>&1 | tee output.log

Darknet does not need to run from within the darknet subdirectory. It is a self-contained application. You can run it from anywhere as long as it is on the path, or you specify it by name.

The various filenames (data, cfg, images, ...) can be relative to the current directory, but I prefer to use absolute filenames in all the darknet configuration files to remove any ambiguity. For example, given the previous "animals" project, the content of the animals.data file might look like this:

classes = 17 train = /home/username/nn/animals/animals_train.txt valid = /home/username/nn/animals/animals_valid.txt names = /home/username/nn/animals/animals.names backup = /home/username/nn/animals

Once training has started, open the image file chart.png to see the progress.

🔗

How should I setup my files and directories?

The Darknet repo is cloned into the ~/src/ subdirectory. Don't put any other personal or modified files into that subdirectory to maintain a clean source directory. This way it will be very simple to "git pull" every once in a while to grab the most recent Darknet changes.

The directory should look like this:

darknet directory

All of the trained neural networks are stored in a completely different subdirectory called "nn". Each project is a different subdirectory from within "nn".

For example, in this image I have 4 custom neural network projects:

neural networks subdirectory

Let's use the "fire" project as an example.

Within the /home/stephane/nn/fire/ subdirectory, I create one or more image dataset directory, and I also create the .names file.

My "fire" project then looks like this:

neural networks subdirectory

Lastly, the rest of the darknet configuration files are also written out to this project directory. (Not the Darknet source code directory!) This includes the .cfg, the .data, and the various other files created by darknet and DarkMark.

Note how I use absolute paths in the .data and various .txt files to remove any ambiguity as to where the files reside.

neural networks subdirectory

Once everything is setup this way, I call darknet directly to begin training, and I continue using absolute path names. For example:

cd /home/stephane/nn/fire/ /home/stephane/src/darknet/darknet detector -map -dont_show train /home/stephane/nn/fire/fire.data /home/stephane/nn/fire/fire.cfg

On Windows the instructions above remain more-or-less the same but replace all instances of /home/stephane/nn/fire/ with C:/Users/stephane/Desktop/nn/fire/.

🔗

What about the "train" and "valid" files?

Contrary to what most people would initially guess, the valid=... line in the .data file is not used for training the neural network! The file is only used to draw the results in chart.png. So especially if you have very few images in your dataset, go ahead and use 100% of the the annotated images for training, and the same 100% for validation.

The conversation with AlexeyAB on this topic was as follows:

Are the images listed in the "valid=..." list used to help train the neural network? Would the following neural networks be identical if this was the only thing different between them: 1) no "valid=..." line in the .data file 2) the "train=..." and "valid=..." files have exactly the same content 3) the "train=..." and "valid=..." files have different content, say a 80/20 split of the annotated files.

The answer from AlexeyAB was:

No, "valid=" isn't used for training and doesn't help train the network. In all 3 cases the accuracy will be the same.

🔗

How to run against multiple images, and/or how to get JSON output?

The preferred way would be to use the API. This way you load the network once, run it against as many images you need, and process the results exactly the way you want.

Darknet has a C API, C++ bindings, and there are other open-source libraries such as DarkHelp which provide an alternate C++ API.

If you are looking for something you can run from the CLI without writing any C or C++:

To use the darknet command line instead of the API, search for the text "list of images" in the Darknet readme. It gives a few examples showing how to process many images at once to get the results in JSON format.

Similarly, DarkHelp also has a JSON/CLI mode which may be used to process many images at once.

🔗

Should I crop my training images?

No!

Say you want a network trained to find barcodes. If you crop and label your training images like this:

barcode

...then your network will only learn to recognize barcodes when they take up 100% of the image. It is unlikely you want that; if the objects to find always took up 100% of the image, then there is little use to train a neural network.

Instead, make sure your training images are representative of the final images. Using this barcode as an example, a more likely marked up training image would be similar to this:

book and barcode

See also: Darknet & DarkMark image markup.

🔗

How to annotate images?

TLDR summary:

Example video that shows how to use DarkMark to annotate images for use with Darknet/YOLO:

Images that don't contain any of the objects you want to find are called negative samples, and they are important to have in your training set. When marking up your images, the negative samples will have a blank (empty) .txt file, telling Darknet that nothing of interest exists in that image.

In DarkMark, this is done by selecting the "empty image" annotation.

empty image in DarkMark

Meanwhile, the rest of your images should have 1 annotation per object. If an image contains 3 vehicles, and "vehicle" is one of your classes, then you must markup the image 3 times, once for each vehicle. Don't use a single large annotation that covers all 3 vehicles, unless that is what you want the network to learn. And similarly, don't break up your object into multiple smaller parts to try and get better or tighter coverage. You should stick to the rule "1 object = 1 annotation".

3 cars

This becomes much more challenging when trying to detect things like clouds, smoke, fire, or rain: the goal is not to cover the image with many small annotations to achieve 100% pixel coverage. Instead, you want to identify each distinct object you'd like the neural network to identify.

Image annotated incorrectly:

incorrect way to annotate images

Same image annotated correctly:

correct way to annotate images

Additional markup comments and techniques are discussed on DarkMark's "markup" page.

See also: How many images?

🔗

How do Darknet/YOLO annotations work?

Darknet/YOLO annotations are saved to text files, one per image. Each line is a bounding box within the image. The annotation coordinates are normalized, and look like this:

0 0.1279690000 0.2870280000 0.0390805000 0.0500511000 1 0.2132137000 0.2787434000 0.0685726000 0.0825132000 0 0.5547320250 0.2944417333 0.0305359500 0.0444498667 9 0.6320830000 0.2855560000 0.0625000000 0.0777778000

The five fields are:

  1. the zero-based class index
  2. the center X coordinate
  3. the center Y coordinate
  4. the width
  5. the height

Here is an example to demonstrate. Say we want to annotate the "5" mailbox digit in this image:

Knowing the center of the "5" is located at coordinate (240, 99.5) and the image dimensions are 560x420, the values are normalized like this:

The classes for this neural network are the digits from "0" to "9", so the first value at the start of the line would be "5". The annotation in the .txt file would look like this:

5 0.428571429 0.236904762 0.078571429 0.097619048

🔗

How do I turn off data augmentation?

If you are using DarkMark, then set to zero or turn off all data augmentation options.

If you are editing your configuration file by hand, verify these settings in the [net] section:

saturation=0 exposure=0 hue=0 cutmix=0 flip=0 mixup=0 mosaic=0

🔗

How important is image rotation as a data augmentation technique?

Depends on the type of image. Some things don't make much sense rotated (e.g., dashcam or highway cam images). But the impact of rotated images needs to be considered. For example, here is a network that is really good at detecting animals:

puppy upside down puppy

With 100% certainty, that is a very cute dog. But when the exact same image is rotated 180 degrees, all of a sudden the neural network thinks this is more likely to be a cat than a dog.

See also: Data Augmentation - Rotation

🔗

Where can I get more help?

Come see us on the Darknet/YOLO Discord!

🔗

TLDR (troubleshooting list)

  1. Do you have the right repo cloned? You should not be using the original one by Joseph Redmon as that one was abandoned many years ago. Instead, please see the popular fork maintained by Hank.ai.
  2. Running into problems building the source code? Building on Linux should be easy. Building on Windows is more complicated.
  3. If training abruptly quits:
    1. Have you checked your paths? Both in the .data file, as well as in the train.txt file? Make sure the images listed are correct.
      (At the same time, may as well verify the .txt annotations are in the same directory as the images.)
    2. What command are you running to train? It should look like this.
    3. Are you running on a headless server? Remember to use -dont_show so Darknet knows to skip opening a GUI window. See the previous question.
    4. Are you running out of memory? Do you see CUDA Error: out of memory in the output?
    5. Curious to see how much vram it will take to train a network?
  4. If the results aren't great:
    1. Are you training for long enough? You should be training for a minimum of 6000 iterations. If you have more than 1 class, then you should be training for a minimum of "2000 * number of classes". So a project with 10 classes will need to train for at least 20,000 iterations. If you have a large number of images, the minimum iterations may be higher. This is explained in the readme.
    2. How many images are you using? If the answer is less than a thousand images per class, then you better know what you are doing.
    3. How many negative samples did you include in your images? If it is significantly lower than half your images, then you need to explain why.
    4. Are you attempting to train using a CPU and had to stop training early?
    5. Having trouble finding small objects? Do you understand how network dimensions, image size, and object size matters?
    6. Do you have very large images, or is the aspect ratio causing problems? In addition to the above question related to network/image/object dimensions, you may also want to consider image tiling, which is available with both DarkHelp and DarkMark.
    7. If everything listed above checks out, but the results are not great, the next step is to verify your annotations. I recommend DarkMark, a free open-source tool to manage Darknet neural network projects, annotate images, verify annotations, and create the necessary Darknet config files.
  5. Darknet CLI usage:
    1. The README.md has examples of the various commands you can run.
    2. Note you can run Darknet against a list of images and get the results in bulk.
  6. Other:
    1. Please see the full FAQ.
 
Creative Commons License
Last modified: 2023-05-22
Stéphane Charette, stephanecharette@gmail.com
../