problem in data recording from custom/html items

[johann.jacoby]

hello,

if this has been discussed elsewhere in the forum I did not find it after an extensive search. i have found a solution to this conundrum now maybe this can save someone else who wrestles with the same thing some trouble.

when using an html file in medialab and collecting several variables from a form in that file (e.g., using dummy variables), medialab kept recording a null character at the end of the value of the last variable in the form (regardless of which variable is last in the form definition, and regardless of where it is written in the data file depending on its alphabetical name). this happened for both text/csv files as well as spss files. the result was a royally screwed up spss file with funny characters in it and text files of which i had to always clean out these null characters first before being able to read them into, say, R for analyses. it seems that this null character originates from the post data that medilab intercepts from the html file and then further processes. these null characters show up in a decent editor (notepad++) as marked in this image:

864

i have now found out that the the problem was with the internet explorer (who would have thunk?) running on my own and our lab machines, which displays the html pages within medialab. these ies used a special character encoding as a default encoding leading to these null characters, for i finally got rid of the null characters in the recording of the data when i explicitly indicated another character encoding for the form alone or in the head of the html file, respectively, like so:

form:
<form name="form1" method="post" accept-charset="ISO-8859-1">or
head:
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">so if you're getting weird data recording check if your ie uses f%§$"&d up non-standard character encodings and make it explicit that you do not want that. in the form or for the entire html file.
possibly try out different encodings until the data recording goes as smoothly as mine does now.

may some deity strike down upon windows' own, non-standard character encodings!

best,
johann

[Jan]

Thanks for figuring this out.
Danke Johann!

[Sven]

I seem to have the same problem. R stops reading my csv data at the strange character. The proposed solution however did not help in my case. I tried character set definitions iso-8859-1, windows-1252, utf-8 and us-ascii without success. I am using IE 7.

Any other ideas?


Edit: As a workaround, I now use GNU sed to remove the characters. In case someone has the same problem and uses R, here's how I do it (be sure to adapt the paths):

mydataframe <- read.csv(pipe("C:/Programme/GnuWin32/bin/sed.exe -e \"s/\\c@//g\" \"C:/path/to/questionnaire.csv\""), header=TRUE, sep=",", quote="", stringsAsFactors=FALSE, comment.char="", fileEncoding="latin1")

I still think it would be preferable if MediaLab would take care of clean data files.

[johann.jacoby]

I seem to have the same problem. R stops reading my csv data at the strange character. The proposed solution however did not help in my case. I tried character set definitions iso-8859-1, windows-1252, utf-8 and us-ascii without success. I am using IE 7.

Any other ideas?

in my original troubles with this, i finally resorted to delete these null characters using a decent editor, i think it was notepad++ [<http://notepad-plus-plus.org/download>]. if i remember correctly, it allows to search [using <\0>] and replace [by nothing] the null characters.
notepad++ is a powerful free software (free as in beer and speech) for windows. so you would have to do search and replace by hand in any data file you have. batch procedures may make that easier if you have many data files.
in general (and especially under linux) you should be able to do this search & replace, possibly with some regular expression trickery or so, in any solid text editor. it is quite the pain in the butt to do it the first time, but i found - even without remembering exactly how i did it in a particular instance, that it will come back very quickly next time around.
make sure you try it out on a copy of a data file first and then if it works alright go on to the original files to prevent data loss.

hope this helps.

johann