Performance issue with many emails?

Note: Translate by Google.

First, congratulations on your work. Really great. Thank you.

Mime_Parser I use a script that I am preparing to process emails eml and enter data in DB.

When I tested with a bulk (> 3000 e) the processing time stretches irregularly.

Results:

PC: Intel Core 2 Duo 2.40 GHz @ 6600
RAM: 2 GB
S.O.: Windows 7 Ultimate (x64)
Posts: 3194, size 1279 Mb, 1558 Attachments (897 Mb).
Apache 2.2.8
PHP 5.2.6.

First Test. Time 1:17:32 process

Second Test. Processing the mail in batches of 1000.
Post 1 to 1000. 360 Mb and 245 Mb 594 Attachments Processed in 0:10:37. In the first test took 0:10:12.
Correos 1001 to 2000. 259 Mb and 183 Mb 348 Attachments Processed in 0:09:36. In the first test took 0:22:20. What is a lag of 00:12:44
Post from 2001 to 3000. 498 Mb and 355 Mb 551 Attachments Processed in 0:10:23. In the first test took 0:35:57. What is a lag of 00:25:34 and a cumulative of about 38 minutes
Post from 3001 to 3194. 162 Mb and 114 Mb 65 Deputy Accused in 0:01:05. In the first test took 0:08:59. What is a lag of 0:07:54 and an accumulated about 46 minutes.

Is this problem may stem from mime_parser or some kind of saturation of Apache or PHP?

Have you been advised of any similar case?

Thank you very much.

Original Text:

En primer lugar, enhorabuena por tu trabajo. Realmente magnífico. Gracias.

Utilizo Mime_Parser en un script que estoy preparando para procesar correos eml e introducir los datos en BBDD.

Cuando hago la prueba con una carga masiva (>3000 correos) el tiempo de proceso se alarga irregularmente.

Resultados:

PC: Intel Core 2 Duo 6600 @ 2,40GHz
RAM: 2 GB
S.O.: Windows 7 Ultimate (x64)
Correos: 3194, Tamaño 1279 Mb, 1558 adjuntos (897 Mb).
Apache 2.2.8
PHP 5.2.6.

Primera Prueba. Tiempo proceso 1:17:32

Segunda Prueba. Procesar los correos en lotes de 1000.
Correos 1 a 1000. 360 Mb. 594 Adjuntos y 245 Mb. Procesados en 0:10:37. En la primera prueba tardó 0:10:12.
Correos 1001 a 2000. 259 Mb. 348 Adjuntos y 183 Mb. Procesados en 0:09:36. En la primera prueba tardó 0:22:20. Lo que supone un desfase de 00:12:44
Correos 2001 a 3000. 498 Mb. 551 Adjuntos y 355 Mb. Procesados en 0:10:23. En la primera prueba tardó 0:35:57. Lo que supone un desfase de 00:25:34 y un acumulado de unos 38 minutos
Correos 3001 a 3194. 162 Mb. 65 Adjuntos y 114 Mb. Procesados en 0:01:05. En la primera prueba tardó 0:08:59. Lo que supone un desfase de 0:07:54 y un acumulado de unos 46 minutos.

¿Este problema puede tener origen en mime_parser o en algún tipo de saturación de Apache o PHP?

¿Te han avisado de algún caso similar?

Muchas gracias.

It is hard to tell because you are performing different heavy tasks.

Parsing large messages is not a light task, but added the message information may be much heavier.

I suggest that you try the same script to parse the messages but do not add the information to the database. That way you will have an idea of what is the weight of the MIME parser class.

The class can be optimized a bit but I am not sure how much performance can be gained.

You can also try a profiler like XDebug to identify the parts that your script spends most time. That way it becomes easier to figure which code segments it would be worth optimizing.

Thanks for your advice. Finally I went through my code. There was a line that slowed the whole process when there was a lot of volume of attachments.

"Gracias por tu consejo. Finalmente repasé mi código. Había una línea que ralentizaba todo el proceso cuando existia mucho volumen de adjuntos"

The class is very fast -- but it does go very slow with larger emails (10mb+... for me at least)

Sorry for the delay. I am catching up on pending messages.

I have not had the time to profile the class and see if are there any points it would be worth optimizing to increase the overall performance noticeably.

If anybody is willing to do such an effort of profiling and determine the sections of the code worth optimizing, just let me know and I can provide guidance on how to do that while I am not able to find time to do it myself.