Hi,

I use OpenNetCF.IO.Port class in my data collecting application running on windows ce with the data stored using SQLCE 3.1 on a CF card. The main application have multiple threads running for collecting data from serial port, inserting the data into the database and also launch another executable upon user request which export the data from the database into a bunch of text files in removable media (SD card/USB thumb drive).

When the second executable is not running, so the application only runs the data collecting and database insert, the application runs fine, there are no serial port buffer overrun reported.

But as soon as the second executable is running which took 100% CPU load doing database reading and writing into text files, the main application started to get buffer overrun error.

I have played with the text files writing thread priority and set it to below normal priority, but it does not help much, lowering the priority reduce the amount of errors but the buffer overrun error still remain.

The only solution I got right now is by putting Thread.Sleep(0) command after each row read from database but this approach really hurt the data export performance.

Is there anyway we can prevent buffer overrun error under heavy I/O load? I'm using .NET CF 2.0, the serial port is set at 19200 baud rate, 7 bit data, 1 stop bit, and odd parity. The hardware that send the data doesn't have handshake capability. Also the database file can be quite large, in the most extreme case I need to export more than 3 million rows of data into text files.

Thanks in advance,
 

Luthfi

Your problem is likely that you're doing the insert on the same thread as the serial read so when you go to do the insert, data keeps coming in and it overruns waiting for you.

 

I'd split this into two threads - one that does nothing but service incoming serial data and one that does the writing.  The incoming serial data I'd put into a Queue and the writing thread would dequeue data from it for inserts.  This would allow the inserts to fall behind the data receive without a problem - you'd still receive data as fast as you can and the  writing thread would finish when it can.

Hi Chris, 

Thanks for the reply. That is exactly how I'm doing it right now, there is already a thread for collecting the data into a queue and another thread for inserting the data. It all run well until the exporting process which runs in another executable is launched.

This second executable also use a separate thread for reading data from db and writing it to text files, the thread is running with below normal priority, any priority higher than that will cause the db inserting thread in the main executable to fall behind badly and eventually will caused an out of memory exception.

To me it seems that file writing/flushing in CE is causing other threads to be blocked momentarily and caused the overrun problem, is this the correct assumption? Does file flushing in CE also use interrupt request? If it does then what happen when two interrupt requests from serial port and from file I/O happened at the same time?

Sure, if you're writing to a slow media like flash I could see it being a problem.  The default quantum for a thread in CE is way longer than you'd want if your serial receiver thread is at the same priority as the writing thread.  Writing could potentially take tens of milliseconds, and if your receive thread has to wait for it to complete before a context switch happens, then yes, I can see data loss as a distinct possibility.  I'd raise the priority of the receiver thread to something like 248 (and avoid doing *any* allocations in that thread - ever).

Ok, so I guess I need to move the message splitting and checksum validation code from the receiver thread and leave only the raw data queuing routine there.

Also I'm using C# on .NET CF 2.0, I see that the Port.cs comm event thread already using Highest priority, I don't know how to set it to anything higher than that. Also I noticed that the dwMaxRxQueue in port capabilities is set to 16 bytes on my device, is there anyway to set this value on the serial port perhaps to some less or greater value?

Hi Luthv

I am experiencing exactly the same overrun problem as you described here, did you manage to get a solution to it?

Thank you,

Philip Fourie

 The "solution" is to not rely on the Port class to buffer all of your data.  Prudent coding would create a background thread that looks for incoming data events and pull any data immediately off and into a local buffer whos size you control (I'd use a Queue<byte> variable).  When the thread adds data to the buffer, your data handling routine should then use that buffer as its data source.

Hi ctacke

Thanks for the feedback.  We are using is the OpenNETCF.IO.Serial.GPS class.  From what I can tell it is already doing what you suggesting, it subscribes to the Port class and then de-queueing the buffer as quickly as it can.

This generally works fine, however when we start pushing the processor (by navigating faster through our application forms) it seems that the Port + GPS classes are falling behind and result in the 'overrun'.

 For this scenario 3 threads are active at the same time:

Thread 1 (highest priority) - Created by the Port class that listens to data coming from the serial port
Thread 2 - Created in the GPS class
Thread 3 - The main GUI thread

Is it possible that the GUI thread is starving Thread 1 from resources and thereby causing the 'overrun' to occur?

Thanks again,

Philip

Hi, The solution for my problem was to increase the priority of the serial port at the driver level by changing some value on the registry since it cannot be done at application level, so I suggest if you have already implement what has been suggested on this thread then you should consult your device manufacturer on solving the problem, my guess is that your display driver has higher priority than the serial port driver. Good luck, Luthfi

Hello Luthfi,

At time of posing my original question the code already contained ctacke's suggestions, leading me to believe that something else is going one - as confirmed by yourself.  Interestingly enough we received new firmware from the device manufacturer a couple of days ago and this has resulted in reducing the overruns dramatically. Leading me to believe that is most likely related to a driver problem... (At the very least we can explore this avenue now, this has really been a tough problem to try and solve :| )

 Great to get confirmation from yourself that this problem can be driver related.

Thanks for the time answering my question,

Philip Fourie