A week ago I made some modifications to the generic file uploading system
I use in web development.
It is basically a stand-alone PHP web application that has the sole purpose
of storing files. Right now it is dependent on Zend Framework, but in the final
version, it should be easy to install anywhere and it should require minimal
configuration. It should also be secure. I will call it the Media
Repository
One possible use case is when building a classic HTML file upload form with
PHP: after the temporary file is stored on the disk, it should get relayed to
the Media Repository and stored there.
The Problem
In short, the problem is you can't upload large files. Initially, I was
using the Zend Framework HTTP Client to relay the file to the repository.
Problem is – when constructing the POST request to the Media Repository, the
large file is stored as a string inside the Zend_Http_Client class. This is okay
for small files, but it can become a problem for large files. The upload time
and size is limited by the following settings:
- The total amount of memory that PHP can allocate (php.ini
memory_limit)
- The maximum execution time and maximum input time (php.ini
max_execution_time and max_input_time)
- The maximum post size (php.ini post_max_size)
- The maximum file upload size (php.ini upload_max_filesize)
The Solution
- Increase max_execution_time and max_input_time. If your
configuration allows it, it can be done with the ini_set() function,
without having access to php.ini
- Use the cURL extension. to check if your PHP installation has cURL, see How to check if an extension is installed. cURL allows you to
avoid reaching the memory_limit when preparing the request. Using the
CURLOPT_INFILE setting, you can instruct curl to transfer specific
stream, instead of a string. It avoids storing the entire file content to
memory.
- It's also a better practice to use the PUT http verb instead of POST when
posting files as raw data:
$ch = curl_init();
$source_file = 'example.txt'
$upload_url = 'http://example.com/folder/example.txt'
// Open the sourcefile
$readfile = fopen($source_file, 'rb');
curl_setopt($ch, CURLOPT_URL, $upload_url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_PUT, true);
curl_setopt($ch, CURLOPT_INFILE, $readfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Relay the file to http://example.com/
$response = curl_exec($ch);
// Close the source file
fclose($readfile);
// close cURL resource, and free up system resources
curl_close($ch);
On the other side of the problem, to recieve the transmission on the Media
Repository, again you have to avoid storing the whole content in a variable.
This time, the POST input can be retrieved by opening the php://input
stream:
$read_handle = fopen('php://input', 'rb');
// The filename where the input must be stored.
$target_filename = 'example.txt';
// Open for reading and writing; place the file pointer at the beginning of the file and truncate the file to zero length.
// If the file does not exist, attempt to create it.
$write_handle = fopen($target_filename, 'wb');
if (($read_handle == false) || ($write_handle == false))
$success = false;
while ($success && !feof($read_handle))
{
$chunk = fread($read_handle, 4096);
$success = $success && (fwrite($write_handle, $chunk) !== false);
}
$success = $success && fclose($write_handle);
$success = $success && fclose($read_handle);
So this I think should be helpful when dealing with large file uploads
in PHP.
POST#0057 2009-FEB-7
Add Comment
Why is respecting deadlines important? Finishing what there has to be done on
time allows us to rely on each other. In my case, deadlines have sometimes been
a problem, mainly because I tried doing way too many things at once. This is
because I sometimes start things without taking into account the time required
to complete that task.
I believe that, in order for deadlines to be kept, there has to be at least
some external motivator – a consequence for failing. There always are
consequences, of course, but in addition to that I'm going to implement the
following plan…
Here's the plan:
In cases where money are involved (because that's what I'm mainly talking
about), a reward for finishing on time (or sooner) must be applied when:
- Finishing very early, in 20% less time: 10% of income allocated to pleasure
spending
- Finishing in 0–20% of time: 5% of money allocated to pleasure
spending
And also a penalty for being late:
- 0–20% more time: 5% penalty
- 20% more time: 10% penalty
What happens to penalty money? Well, it goes back to the customer, or, in
other cases, it goes to a cause I don't like or disagree with (the Church for
instance).
Starting from now on, this is how I plan to become better at not missing
deadlines and finishing stuff on time.
POST#0056 2009-JAN-16
Add Comment
Alice is a brand owned by Telecom Italia, one of the largest fixed phone
operator in Italy. Alice is dedicated to offering ADSL Internet, also servicing
Germany, France, San Marino and the Netherlands (http://en.wikipedia.org/…lecom_Italia).
The Alice
Gate 2+ is the default ADSL modem you receive after installing an ADSL line
for the Internet in Italy. One model has a Wi-Fi antenna and some ports you can
use to connect devices via cable. There is also a version featuring a VoIP
service which you can use instead of the primary land-line phone, but forces you
to change your phone number.
After the physical installation of the modem, there are a few problems you
could possibly run into. Depending on the operating system used, you can have no
Internet connectivity at all or you can have problems with file uploads and
downloads.
Fortunately, they're quite easy to solve. The way to go is to edit your
operating system tcp settings.
Ubuntu, Linux, Mac OS X
Symptom: after a few sent/received bytes, the network traffic stops. To fix
this, open /etc/sysctl.conf as root (create it if it doesn't
exist). Make sure it contains the following values:
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_ecn= 0
Windows Vista
In Windows Vista, it's impossible to upload files and sometimes downloads
stop after the first few kilobytes. If this is what's happening to you, open a
new Command Prompt as Administrator and see the current tcp configuration:
netsh interface tcp show global
Then disable the autotuninglevel and ecncapability:
netsh interface tcp set global autotuninglevel=disabled
netsh interface tcp set global ecncapability=disabled
This will probably help you fix the problem with this modem, but you can find
more about it at the source I used for this article: http://www.lemiefrequenze.org/…archives/162
POST#0055 2009-JAN-11
Add Comment
After a very busy period, I finally have had the time to fix some pending
issues with the blog and here I am writing the first post this year.
How have I been, you ask? Here's a summary:
- The Mondenii (fashionable, high
life) is a very popular and quite entertaining TV Show in Romania, laughing at
politicians, celebrities and trends such as „Emo“. I have had the pleasure
of working on the Mondenii website back-end (and a little bit on the front-end)
with the Adviser Interactive team, Alex and Ionut.
- Moved to Italy, spent Christmas and New Year here. Italians are great people
and they have large hearts, at least most of the ones I've met. But they are
likely to be reserved about expenditures! Except for my friends, so far I miss
Romania not a single bit.
- The web is not often part of the typical Italian life. It looks like they
are quite focused on relationships and friends. But the social networking sites
caught up pretty well here (as they did everywhere in the world) and they also
use Skype for free video and audio calls. Other than that, they may be the most
technologically backwards people in whole Europe!
I have also started work on a possible start-up idea involving tutorials and
wiki websites while looking for collaboration/work near Turin or Milan.
POST#0054 2009-JAN-10
Add Comment
If you have to get to a place you've never been before, are you the kind of
person that never asks for directions, but rather relies on signage and
planning?
„Apparently Google assumes you're traveling during the ferry's normal
operating hours. We lost two hours circling that damn lake (to say nothing of
the Straw Man).“ – http://xkcd.com/461/
POST#0053 2008-AUG-11
Add Comment
Pentru Windows am găsit o soluţie acceptabilă la problema
din articolul precedent, distribuirea codului sursă modificat la un anumit
site cât mai comodă.
Atunci când e nevoie de un transfer dintr-ul loc de pe web pe altul, am
început să folosesc WebDrive, care
mapeaza o locaţie pe web (FTP, WebDAV şi încă câteva) pe un drive Windows
în My Computer.
Se pot face mai multe conexiuni în acelaşi timp, adică mai multe
drive-uri.
În final rulez un fişier .bat care copiază cu xcopy directoarele
relevante de pe un drive pe altul. Fişierul .bat poate să facă parte chiar
din repository. Eh tot trebuie scris un pic de fişiere .bat dar e mult mai bine
decât înainte. O să revin probabil cu mai multe detalii după ce rafinez
puţin procesul.
Sper ca această informaţie să fie de ajutor şi altcuiva :) Poate există
şi alte sisteme mai bune, voi căuta în continuare.
POST#0052 2008-MAR-21
Add Comment
Eficientizare
Dezvoltând aplicaţii în limbajul PHP, orice dezvoltator ajunge la un punct
în care de fiecare dată când clienţii doresc o modificare minoră a unui
site, se pierde un pic de timp corectând problema dar mai mult timp
actualizând conţinutul de pe serverele web.
Dacă ar exista o metodă de upload universală la un click de mouse, ar fi
extraordinar pentru că s-ar câştiga timp preţios şi viaţa tuturor ar fi
mai fericită. Clienţii ar putea fi taxaţi mai puţin pentru operaţiile de
întreţinere şi profitabilitatea ar creşte.
Sunt în căutarea unui instrument orientat către dezvoltatorii de PHP care
să rezolve următoarele probleme…
Transfer al aplicaţiei pe server
Partea care îmi place cel mai mult la PHP este că lansarea aplicaţiei
constă doar în plasarea fişierelor într-un director. Nu este nevoie de nici
o compilare, totul este simplu. Unde se află codul sursă al aplicaţiei la
început:
- În sistemul de fişiere
- Într-un sistem de version control (ex: Subversion).
Vrem să-l punem pe server:
- Prin FTP, de exemplu pe shared hosting.
- SSH (SCP, SFTP).
Fişierele de configuraţie
În PHP pentru definirea setărilor (datele de conectare la baza de date,
e-mail, debugging…) fiecare foloseşte propria metodă:
- XML, care permite editarea valorilor uşor de către un script
- Bazat pe un fişier de configurare .php, în funcţie de mediul de lucru
(producţie/testare)
- Etc..
Baza de date
Atunci când modificăm structura unei baze de date, ar fi grozav dacă s-ar
putea aplica schimbările şi pe server. Aici apare o problemă totuşi, pentru
că nu putem face un dump întreg al bazei de date cât timp baza de date remote
conţine date importante.
- Pentru scopuri simple, s-ar putea folosi aceeaşi bază de date pentru teste
ca cea pentru producţie.
- Baza de date pentru teste diferită de cea live, caz în care sincronizarea
va trebui făcută manual.
În cazul meu s-ar putea presupune că modificările din baza de date sunt
atât de rare încât nu necesită automatizare.
Soluţia definitivă
Am auzit numele Capistrano, dar presupune o familiaritate cu ruby. Necesită
ca acesta să fie instalat pe sistem.
Eu simt lipsa unei aplicaţii care să poată fi folosită pentru
sincronizare. Ar putea consta în câteva tool-uri pentru command line şi
o interfaţă grafică care să le apeleze. Ar putea face transferul către
server şi ar putea interacţiona cu repository-uri. Eventual ar fi extensibilă
prin module.
Soluţia definitivă eu nu am găsit-o. Presupun că există mulţi alţi
dezvoltatori în această situaţie şi mă întreb cum ar fi dacă am avea
acest software.
POST#0051 2008-JAN-30
Add Comment
După o pauză mai lungă de scris prin blog, am decis să mai fac câte
ceva. Principala problemă era cauzată de spam, am şters cateva mii de
comentarii cu link-uri aiurea postate de spammeri.
Cu noua metodă de adaugare a comentariilor vreau să reduc numărul de
mesaje nevrute la 0, şi sper că vizitatorii se vor simţi comfortabil să
posteze chiar şi aşa…
Oricum, în curând şi ID-urile de Yahoo vor putea fi
folosite pentru a posta comentarii aici.
POST#0048 2008-JAN-18
Add Comment
Pentru cei care urmăresc starea zăpezii în anticipaţia unei drumeţii de
schi/snowboard, iată câteva surse pentru a afla care e starea stratului:
Ar trebui să stau la calculator şi să muncesc dar aş vrea să merg luni
– imediat după weekend – ca să fie cât mai libere telescaunele şi tot
– probabil la Sinaia.
POST#0047 2007-DEC-8
Add Comment
Sunt foarte recunoscător celor care au subscris la blog-ul acesta şi regret
faptul că feed-ul a fost stricat în ultima perioada. Tocmai l-am reparat şi,
după ce se curăţă cache-urile de prin readere şi agregatoare ar trebui să
fie OK! Mersi pentru ajutor la depistarea problemei.
POST#0046 2007-NOV-16
Add Comment
