FC++  v0.9.0-9e9b65
FileCatalyst Fast File Transfers - C++ Library
Getting Started

Understanding FC++ Parameters

The command-line tool fccl (or fccl.exe on Windows) uses the same FC++ library that is shipped as part of the FC++ SDK API. Prior to attempting to use the FC++ SDK API in your application, it is strongly recommended that you try a file transfer with fccl to understand the different file transfer options and their impact.

To get started, this simple command will download a file from a FileCatalyst server:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip

While this is enough to get started, it is not the most optimal transfer. Let's add a few additional parameters to understand what happens during a FC file transfer.


–showstats

The –showstats parameter will display some additional statistics after the transfer has completed. We'll use those statistics to understand the impact of several other parameters.

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats
Downloading "backup.zip" over UDP:
/backup.zip:
Local filename: ./backup.zip
Remote filename: /backup.zip
Unit size: 1.00 KiB
Block size: 2.50 MiB
Padding size: 1 B
Transfer rate: 1.12 Mbps
File size: 5.00 MiB
Blocks completed: 2
Packets received: 6172
Packets sent: 139
Duplicate packets: 913 [15.13%]
Retransmit requests: 1452 [24.07%]
ACK packets: 19
Data packets: 6033
Echo packets: 60
Metrics received: 60
Metrics sent: 60
Transfer type: full file transfer
Time elapsed: 0:37

 

Field Meaning API reference Formatting
Local filename name of the file on the FC++ client computer fc::TransferStats::localFilename  
Remote filename name of the file on the FC server computer fc::TransferStats::remoteFilename  
Unit size size of payload in each UDP packet –blocksize and –unitsize
fc::TransferStats::unitSize
fc::formatBytesIEC()
Block size units are combined to create file blocks –blocksize and –unitsize
fc::TransferStats::blockSize
fc::formatBytesIEC()
Padding size number of "empty" bytes added to the last block –blocksize and –unitsize fc::formatBytesIEC()
Transfer rate average transfer rate fc::TransferStats::bitsPerSecondAverageTransferRate
fc::TransferStats::bitsPerSecondImmediateTransferRate
fc::formatTransferRate()
File size file size in bytes fc::TransferStats::payloadBytesExpected fc::formatBytesIEC()
Blocks completed file blocks transferred –blocksize and –unitsize
fc::TransferStats::blocksCompleted
fc::TransferStats::blocksRemaining
 
Packets received total number of data and non-data packets received fc::TransferStats::packetsReceived  
Packets sent total number of data and non-data packets sent fc::TransferStats::packetsSent  
Duplicate packets unecessary duplicate packets that had to be dropped –congestioncontrol
fc::TransferStats::duplicatePacketDropped
 
Retransmit requests number of packets that had to be re-requested –congestioncontrol
fc::TransferStats::requestRetransmit
 
ACK packets number of ACK packets fc::TransferStats::ackPacketsReceived  
Data packets number of payload data packets fc::TransferStats::dataPacketsReceived  
Echo packets number of echo packets fc::TransferStats::echoPacketsReceived  
Metrics received number of metric packets received fc::TransferStats::metricsPacketsReceived  
Metrics sent number of metric packets sent fc::TransferStats::metricsPacketsSent  
Transfer type how the file was transferred fc::TransferStats::type
fc::TransferStats::EState
 
Time elapsed the time it took to transfer the file fc::TransferStats::timeRemaining
fc::TransferStats::timeTransferBegan
fc::TransferStats::timeTransferFinished
fc::formatTimeDuration2()
See also
fc::Control::getMostRecentFileStats()
fc::TransferStats::getStr()

–bandwidth and –startrate

One of the first things we normally want to control is the speed at which a transfer will take place. This is done with the –bandwidth parameter:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500

However, if you try this on a large enough file, or across a busy network such as the internet versus an internal network, you'll note you didn't get the rate you requested. This is because FC++ gradually ramps up to the specified speed. If you want to force FC++ to use a specific speed, you'll also want to specify a starting rate indentical to the desired maximum rate:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500
See also
fc::Options::setBandwidth()

–congestioncontrol

Part of the statistics displayed when –showstats has been specified lists the number of duplicate packets as well as the number of re-requested packets. Duplicate packets are not a big concern unless the number is extremely high. But re-requested packets indicates a problem that can result in poor file transfer performance. The goal is normally to get this number close to 0%. Here is an example of what that output can look like:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500
...
Transfer rate: 0.57 Mbps
File size: 5.00 MiB
Duplicate packets: 569 [10.00%]
Retransmit requests: 1751 [30.78%]
...

The 1751 retransmit requests in this example means the receiver had to ask the sender 1751 times to re-send a different part of the file because it was lost in transit. There are several possible reasons why this happens:

  • The original UDP packets were dropped by the sending host.
  • The original UDP packets were dropped by an intermediate router or QoS device on the network.
  • The original UDP packets were dropped by the receiving host.
  • The original UDP packets weren't read quickly enough from the network stack by FC++.

The easiest way to alleviate this problem is to turn on the FileCatalyst UDP congestion control. This allows the sender to slow down when the receiver detects that some packets have been dropped.

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol
...
Transfer rate: 0.96 Mbps
File size: 5.00 MiB
Duplicate packets: 31 [0.60%]
Retransmit requests: 31 [0.60%]
...
Note
UDP congestion control also has an optional –aggression parameter to indicate how quickly it should try to recover when dropped UDP packets have been detected.
See also
fc::Options::setCongestionControl()

–multithread and –numencoders

By default, UDP download uses a single thread within FC++. This helps to limit the amount of memory needed while downloading, but it also limits the speed at which transfers can happen. If too many UDP packets arrive while the single thread is busy processing a previous packet, some of the newly-arrived UDP packets will be dropped by the operating system before FC++ can process them.

This problem can be addressed by enabling multi-threaded downloads. You'll also need to specify the number of threads which FC++ will use. Try with 3 to 5 threads to see if it makes a difference. Having threads also introduces lock contention, so there comes a point where adding more threads doesn't help.

When adding threads, it also helps to increase the number of file encoders on the server. The FC++ API tells the server the number of encoders to use with the –numencoders parameter. It is a good idea to have 1 or 2 more encoders on the server than the number of threads on the FC++ client.

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 6
See also
fc::Options::setMultiThreaded()
fc::Options::setEncoders()

–blocksize and –unitsize

Files transferred using FileCatalyst are broken up into parts called blocks and units. Every block is made up of 1 or more unit, and in typical usage, a single unit also happens to fit within a single UDP packet. The default block size is 4 MiB in size, and the default unit size is 1024 bytes.

The block size is typically not a problem, but the unit size can be further optimized. If you know the MTU between the FC server and the FC++ client, you can increase (or decrease) the unit size. With a typical network MTU of 1500 bytes, and taking into account the overhead that FileCatalyst needs, this means you can put more than the default 1024 data payload into each packet. For example:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 5 -blocksize 262144 -unitsize 1300
...
Unit size: 1.27 KiB
Block size: 138.38 KiB
Padding size: 21 B
File size: 5.00 MiB
Blocks completed: 37
...

In the above example, the block size was decreased from 4 MiB to 256 KiB, and the unit size was increased from 1024 bytes to 1300 bytes. But 1300 doesn't cleanly divide 256 KiB, and the size of the download file also isn't a perfect multiple of 1300. So FC++ automatically reduces both the block size and the unit size as necessary so the file can be transfered with a minimal amount of internal padding. In this example, 37 blocks of ≈138 KiB were downloaded, and a total of only 21 bytes of internal padding was necessary on the very last block.

Warning
There is a limit to the number of FileCatalyst blocks that can be in transit during a file transfer. If your blocks are too small, the overall transfer rate may be negatively impacted since a new block cannot be started until a previous block has finished transferring.
Note
If the transfer is happening on an internal network with support for jumbo packets, increase the FC unit size to take advantage of the increased MTU.
Likewise, if the transfer goes through a VPN, GRE, or other IP tunnel, remember to reduce the FC unit size.
See also
fc::Options::setLimits()
fc::Control::getOptimalUnitAndBlockSizes()

–verify

MD5 checksum verification can be enabled to ensure the transfer wasn't corrupted. This runs after the transfer has finished, and can take non-trivial time to run on large files. By default, MD5 verification is disabled. If needed, it can be enabled using the –verify parameter.

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 5 -verify
See also
fc::Options::setMd5Verification()

–ssl and –port

To encrypt the UDP transfer, SSL must be enabled on the FileCatalyst server. The default port for encrypted FTP is usually port 990. Once the server has been configured, downloading a file requires adding –ssl and –port:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 5 -ssl -port 990
See also
fc::Options::setEncryption()
fc::Options::setFtpServer()

–summary

A log showing both the client-server conversations as well as file transfer statistics can be created with –summary. This can also be useful to debug situations where the login process or initial file transfer isn't working as expected.

See also
fc::Logging::enable()
fc::Logging::EType::kSummary

Summary

Here is an example of how to download a file from the command-line using fccl:

fccl -download -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 5 -unitsize 1300

When using C++, the exact same download can be accomplished using this code:

fc::Options options( "bob", "mypass", "192.168.1.3" );
options.setBandwidth( fc:Mbps( 2.5 ), fc::Mbps( 2.5 ) );
options.setCongestionControl( true );
options.setMultiThreaded( true, 4 );
options.setEncoders( 5 );
options.setLimits( 1300 );
fc::Control ftp( options );
ftp.download( fc::Remote( "backup.zip" ) );

Here is the equivalent command to upload a file:

fccl -upload -host 192.168.1.3 -user bob/mypass -file backup.zip -showstats -bandwidth 2500 -startrate 2500 -congestioncontrol -multithread 4 -numencoders 5 -unitsize 1300

The C++ code is similar to the previous example, except for the very last line:

ftp.upload( fc::Local( "backup.zip" ) );

Additional parameters available to fccl are listed in FC++ API (core), or by running "fccl --help".

See also
Recommendations
fc
Definition: FCControl.hpp:17
fc::Mbps
uint64_t Mbps(const int rate)
Convert easy-to-use numerical values to bps values.
Definition: FCOptions.hpp:271
fc::Options
fc::Options contains all the default and user-defined settings required by fc::Control to connect to ...
Definition: FCOptions.hpp:282
fc::Remote
The class fc::Remote is used to encapsulate remote directories or files.
Definition: FCPath.hpp:160
fc::Local
The class fc::Local is used to encapsulate local directories or files.
Definition: FCPath.hpp:141
fc::Control
Definition: FCControl.hpp:52