Chi Square: Language Recognition II

Chi Square: Language Recognition II

I thought I would build on the last post by making a simple spreadsheet that can then easily show which language is being used.  I chose the groupings of letters such that as long as there are at least 1000 letters in the text it will satisfy the Chi square condition of no expected values less than 1 and no more than 20% of expected values less than 5. 

Data

I asked Chat GPT to write me a 1000 word piece in Italian.  This is the first paragraph:

Mentre passeggiavo per le strade, potevo sentire il profumo dei fiori che adornavano i balconi e le finestre delle case. I colori vivaci dei petali si fondevano armoniosamente con la palette di colori della città. Ogni angolo era una nuova sorpresa, un nuovo dettaglio da scoprire. Era un piacere per i sensi.

I then pasted this into a frequency analysis site here.  This counted the total letters used as 3976 and gave the following result:

I then set up a spreadsheet with the expected frequencies for English, Italian and French.  This then allowed me to use a formula to work out all expected values very quickly.  The only data to then add was the observed values – and then the spreadsheet was able to do the rest.

French:

The data for comparison with French is shown below:

We can see that this returns a very high Chi Square value.

English:

Here are the results for comparison with English:

This returns an even higher Chi Square value.

Italian:

When we compare with Italian we get the following:

So, based on a comparison between Italian, English or French our test strongly supports the likelihood of it being Italian.  Notice that we are not doing a hypothesis test as such here – we are simply comparing Chi Square values.  Also we can note that because we are using nearly 4000 data points we are likely to get Chi Square values greater than significance levels regardless of the fit (this is a weakness in the Chi Square test).

So – once the spreadsheet is set up, this is a pretty quick way of identifying languages that share the same alphabet – with no language skills required!