Hi,

I use OpenNetCF.IO.Port class in my data collecting application running on windows ce with the data stored using SQLCE 3.1 on a CF card. The main application have multiple threads running for collecting data from serial port, inserting the data into the database and also launch another executable upon user request which export the data from the database into a bunch of text files in removable media (SD card/USB thumb drive).

When the second executable is not running, so the application only runs the data collecting and database insert, the application runs fine, there are no serial port buffer overrun reported.

But as soon as the second executable is running which took 100% CPU load doing database reading and writing into text files, the main application started to get buffer overrun error.

I have played with the text files writing thread priority and set it to below normal priority, but it does not help much, lowering the priority reduce the amount of errors but the buffer overrun error still remain.

The only solution I got right now is by putting Thread.Sleep(0) command after each row read from database but this approach really hurt the data export performance.

Is there anyway we can prevent buffer overrun error under heavy I/O load? I'm using .NET CF 2.0, the serial port is set at 19200 baud rate, 7 bit data, 1 stop bit, and odd parity. The hardware that send the data doesn't have handshake capability. Also the database file can be quite large, in the most extreme case I need to export more than 3 million rows of data into text files.

Thanks in advance,
 

Luthfi

Your problem is likely that you're doing the insert on the same thread as the serial read so when you go to do the insert, data keeps coming in and it overruns waiting for you.

I'd split this into two threads - one that does nothing but service incoming serial data and one that does the writing.  The incoming serial data I'd put into a Queue and the writing thread would dequeue data from it for inserts.  This would allow the inserts to fall behind the data receive without a problem - you'd still receive data as fast as you can and the  writing thread would finish when it can.

Hi Chris, 

Thanks for the reply. That is exactly how I'm doing it right now, there is already a thread for collecting the data into a queue and another thread for inserting the data. It all run well until the exporting process which runs in another executable is launched.

This second executable also use a separate thread for reading data from db and writing it to text files, the thread is running with below normal priority, any priority higher than that will cause the db inserting thread in the main executable to fall behind badly and eventually will caused an out of memory exception.

To me it seems that file writing/flushing in CE is causing other threads to be blocked momentarily and caused the overrun problem, is this the correct assumption? Does file flushing in CE also use interrupt request? If it does then what happen when two interrupt requests from serial port and from file I/O happened at the same time?

Sure, if you're writing to a slow media like flash I could see it being a problem.  The default quantum for a thread in CE is way longer than you'd want if your serial receiver thread is at the same priority as the writing thread.  Writing could potentially take tens of milliseconds, and if your receive thread has to wait for it to complete before a context switch happens, then yes, I can see data loss as a distinct possibility.  I'd raise the priority of the receiver thread to something like 248 (and avoid doing *any* allocations in that thread - ever).

Ok, so I guess I need to move the message splitting and checksum validation code from the receiver thread and leave only the raw data queuing routine there.

Also I'm using C# on .NET CF 2.0, I see that the Port.cs comm event thread already using Highest priority, I don't know how to set it to anything higher than that. Also I noticed that the dwMaxRxQueue in port capabilities is set to 16 bytes on my device, is there anyway to set this value on the serial port perhaps to some less or greater value?