|03:51 am - Conlang Evaluation|
Conlang evaluation: a framework for criticism
By Sai Emrys - firstname.lastname@example.org / http://saizai.com
16 December 2006
David Peterson posted an entry in January 2002 later entitled the Conlanger’s Manifesto, an apologia in defense of conlanging as an art to be taken seriously . In March 2002, Jesse Bangs wrote an entry  on the CONLANG mailing list entitled “Lighting Some Flames: Towards conlang artistry”, and later known as the Artlanger’s Rant. It went one step further, advocating for a new body of conlang criticism. This part was relatively well received, except that he also advocated for “schools” of conlanging – his being the Naturalist school – and that triggered a rather… heated debate.
Most of the problems people had with the idea of conlanging schools came under two basic categories:
1. Concern that doing so would split up what is already a relatively small subculture (approximately 1000 members worldwide, of whom probably 100 are publicly active), and set the schools in opposition to each other.
2. Belief that conlang criticism could not be done because the intention of the creator can vary so widely – and that, e.g., applying Jesse’s Naturalist criteria to languages that don’t hold the same values as his would be inapplicable at best or ideologically arrogant at worst.
When I posted questions asking about readers’ *own* priorities in prep for this essay, the response was by contrast completely positive and non-flamewar-like. Perhaps this is from the community having learned from the previous experience (and avoiding those subjects), but I believe it is in what exactly is asked, and what the assumptions behind the questions are.
INTENT OF THIS ESSAY
My intent here is not to reignite that 2002 controversy, but to offer an alternative – one that serves the purposes of Jesse Bangs in giving a framework for and encouraging substantive conlang critique; is value-neutral and takes into account the varying goals of language creators; and is fun and easy to use yet comprehensive.
In my opinion – and from my surveys – there is no doubt that there are numerous criteria that can be applied to examine a given language (whether natural or created). For example, one can ask how well documented it is; how closely its features fit in the bell-curve of natural language ‘universals’; how aesthetically pleasing it is from certain perspectives; how regular its grammar is; how easy it is to learn.
Some of these questions can be evaluated objectively. To rate ‘universality’ for example (see below), take each of its features (e.g. split ergativity) for which there is a known occurrence frequency in natural languages, and compute an average number using those frequencies to say “how normal is this language, compared to all known natural languages”.
Others can only be evaluated subjectively. For example, phonaesthetics will vary widely depending on the reader’s personal preferences, cultural background, etc. At best for this one can try to define some types of aesthetics that are relatively consistent and agreeable (such as “mellifluousness”), and rate the language for that rather than for general subjective “aesthetic” in the “sounds good” or “sounds bad” sense – because tastes can vary so widely.
I must emphasize strongly that none of these criteria have any *inherent* value to them. Some creators may prefer to have a highly regular language; others may prefer one that is just irregular enough to add ‘spice’; still others may actually prefer a language that is so irregular as to be cryptographically baffling.
As such, though each criterion can be evaluated, there is no inherent “value” rating to the criteria – i.e. no concept of “good” or “bad”, merely specific value-neutral qualitative questions. They should be treated solely at face value for each item.
CRITERIA - META
The criteria I describe here can be used as a sort of Conlang Code, à la Geek Code et al. To distinguish from the Conlang*er* Code v1.1, I dub this the Conlang Code v2. They can be used in parallel and mostly do not overlap. 
For each element, ratings are listed. They can be used alone for classification and rating purposes, but for a real review it would of course be more helpful to include a per-criterion breakdown where relevant explaining the reasons for evaluating each item as such, or with suggestions for how to improve it (toward the author’s desired state). They can also be used to state what your goals are for your language(s).
Each criterion is rated on a scale from ---- to ++++, with some other options (using !) for when the answer falls outside that spectrum. Alternatively, criteria can be listed in numeric format for concision. E.g. nat-(+++) = nat-1(3).
Variables work much like the Conlanger Code:
! placed after: not enough data to evaluate (review)
* placed after: item does not apply (review); indifference, neutrality (goals)
[item omitted]: item was not evaluated (review); indifference / neutrality (goals)
() placed after: value varies (e.g. foo++(+++) = foo varies from ++ to +++)
! placed before: tertium quid or other special variation (see descriptions)
Code abbreviations are the capital letters of their heading. (E.g. nat for NATuralness). They are listed in lowercase (“nat”) in the code or as a rating (“nat++”), but in uppercase when referring to the criterion itself (“NAT is a criterion that….”).
Some items have sub-criteria listed under ‘extras’. These are optional.
Most of these criteria can be approached both objectively and subjectively. In a detailed review, one should expect to find for each point the rating followed by specific reasons (possibly empirical ones, possibly not) why that rating was given.
All reviews are *only* expected to be accurate of what is publicly documented as of the date of the review.
The review should normally be of the *primary* mode of the language. In most cases, this will be the spoken one. (See MOD.)
Part 1: CLCv2 (for “ConLang Code version 2” – this part is invariant)
Part 2: Date of writing this code (mandatory!), in DMonYY format
Part 3: Code usage type
R Conlang Review (by author or others)
G Conlang Goals (permissible by author only)
Part 4: Language(s) covered by this code (author may have separate Goal codes for separate languages, or cover multiple ones in one code if they’re similar enough in goals)
Part 5: Priority (goals version only)
++++ I work on this nearly all the time.
++ This is my main current project.
-- Enh, someday I’ll edit it.
---- No longer under development and not likely to be any time soon.
If priority differs, list appended to each language name; otherwise just append to the last one.
E.g. a review might start with: CLCv2 6Dec06 R Foolang …
Or a goals code might start with: CLCv2 23May05 G Erisian-, Discordian++ …
In deference to Jesse, I will start with his "naturalist" school's primary criteria.
NATuralness: Appears to be a normal human Earth language.
nat++++ At the apex of normality: exactly 2.3 children, half a divorce, etc. (Within 0.5 s.d. of mean.)
nat++ Medium normal. No linguist would blink an eye at finding a description of it in Languages of the World. (Within 2 s.d.s of mean.)
nat Within the bounds of normal, but perhaps somewhat quirky. (Within 3 s.d.s of mean.)
nat-- Not really normal at all, but maybe if you squint *really* hard… (Within 4 s.d.s of mean.)
nat---- Just plain bizarre. 10 clicks, split ergativity, and a built-in encryption algorithm? (What mean?)
!nat++ This language is fairly normal… for the world/culture in which it’s set (“Well, if there *were* a blind space faring avian race, this probably *would* be a pretty normal language for their physiology…)
E.g. languages: French: nat++ Esperanto: nat; All Noun: nat---
Goals: Jesse Bangs: nat++(+++); ye generic loglanger: nat*; ye generic sci-fi author: !nat+
ComPLeteness: Can be used to talk about anything.
cpl++++ Shakespeare, astrophysicists, and politicians can all use it.
cpl+++ Can be used to translate Hamlet.
cpl+ Can be used to translate the Babel Text
cpl Enough for tourists to get directions…
cpl-- Well, I’m kinda still working on orthography…
cpl---- I only just named it!
!cpl++ It can be used to talk about most subjects… within the limited scope it’s intended for. (e.g. “Battle-Command-ese”)
E.g. languages: English cpl++++; Toki Pona cpl+(+++); Quenya: cpl+; ye generic toylang: cpl—
Goals: ye generic auxlanger: cpl++++; ye generic sci-fi author: cpl
ComPleXity: Number of rules, exceptions to those rules, and various interacting contradictory levels of grammar.
cpx++++ Might require a doctorate and supercomputer to be able to use in real time.
cpx++ Wait, I thought you said I comes before E and sounds like /i/…
cpx At least it’s no harder than my Spanish class in high school.
cpx-- So there’re a couple small exceptions, but otherwise it’s pretty simple and regular…
cpx---- There’s only a dozen rules? No exceptions? Everything else is free?
E.g. languages: Ithkuil cpx++++; Sanskrit cpx+(+++); Toki Pona cpx----
Goals: ye generic auxlanger: cpx--(---); ye generic artlanger: cpx+(++); ye generic loglanger: cpx*
Note: complexity of rules does not necessarily equate to complexity of output. Viz. John Conway’s /Life/, or the Turing machine.
Personal INnovation: Breaks away from the author’s preconceptions about how language works (i.e. their L1s).
pin++++ You, a non-linguistically-trained native English-only speaker, came up with split ergativity by yourself?
pin++ Playing with bits not present in your L1 language family…
pin---- Another relex, eh.
!pin Sure it’s a euroclone and the author’s L1s are English and French, but zie is 100% aware of all the places zie copied zir native grammar and explicitly decided to do so.
!!pin The author seems to be rejecting their L1 grammar without real thought!
E.g. languages: almost everybody’s first conlang (be honest now): pin---
Goals: my students’ for their final project: pin++
Global INnovation: Something never done before in human language.
gin++++ [insert doctoral thesis and Nobel science award here]
gin++ You have a language… with no nouns!?
gin The language has a unique “feel”.
gin---- Yes, people have made European auxiliary languages before…
!gin+ That’s a pretty unusual language concept… for a sketch / fiction story.
E.g. languages: ye generic IAL: gin----; Heptapod B: !gin+++
Goals: ye generic loglanger: gin+++(++++); ye generic artlanger: gin*()
CoHeRence: Overall fit, consistency of ‘feel’, etc.
chr++++ Every singly aspect of the language is brilliantly tailored to the same goal.
chr++ Everything fits together well, with a very definite “feel”.
chr-- Somebody’s been learning new things in Linguistics class again…
chr---- Design by committee using dice rolls, a random word generator, and every grammatical feature ever mentioned on CONLANG.
E.g. languages: Toki Pona chr+++; my conlang class languages: chr--(---)
Goals: ye generic loglanger, artlanger, and auxlanger: chr++++
CuLTural expressiveness: Shows that it is attached to a specific culture.
clt++++ A sociolinguist could create a complete analysis of the mores, technological level, and societal structure of the culture that uses this language just from looking at the language.
clt++ The language is clearly tied to a strong cultural
clt---- You can’t even tell that such a thing as sociolinguistics exists judging from this language.
E.g. languages: Quenya clt++
Goals: ye generic auxlanger: clt----; ye generic artlang: clt++++
LIBeralness of culture expressed: In the generic American sense.
lib++++ We love everyone! No biases! Completely gender-equal! Sex positive!
lib---- Only the righteous deserve respect.
!lib Does not sufficiently express a culture in the first place.
E.g. languages: Laadan lib++; Newspeak lib---
Goals: ye generic auxlanger: !lib; ye generic feminist conlanger: lib+++
MeLliFluousness: Smooth, pretty, poetic in its phonaesthetic.
mlf++++ The most vile curses sound like lullabies.
mlf---- Spoken by an angry Klingon with a bad cold.
E.g. languages: Quenya mlf++(++++), Klingon mlf---(----)
SaPir-Whorf: Encourages (or forces) a certain worldview through its structure.
spw+++++ Grants new powers – genius, psychic ability, etc. – or permanently changes the speaker’s psychology.
spw++++ Makes it impossible to even think about something against its design.
spw++ Strongly encourages a specific way of looking at the world.
spw---- Exactly the same as speaking any other generic L1.
!spw+++ Does not *require* it, but does *allow* one to think about things nearly inexpressible or inconceivable in a generic L1.
E.g. languages: Heptapod B spw+++++; Toki Pona spw+++(++++); Laadan !spw++
Goals: ye generic cognitive psycholinguist: spw+++;
EASe of learning: Ease with which this language would be learned (as an L2)
eas++++ Anybody could be fluent within days.
eas---- Would require a PhD and 20 years of exclusive study just to ask offhand how to find the bathroom.
!eas++[eu] Qutie easy to learn… if you already know a eurolang. (See below for the two-letter abbreviations.)
Extras: L1 acquisition ease (format: eas[L2 ease]:[L1 ease])
E.g. languages: ye generic African click language eas---:+; Toki Pona eas++; Ithkuil eas---
Goals: ye generic auxlanger eas+++(++++); ye generic euro-auxlanger !eas+++(++++)[eu]; ye generic a priori loglanger eas*:++
DOCumentation: Thoroughness of the *publicly available* documentation.
doc++++ Dictionaries, linguistic analysis, primers – the works. And all easily understandable.
doc++ Thorough description available, enough to learn the language if you look.
doc Enough to at least get a pretty good feel for what it’s all about
doc-- More of a sketch than an actual documentation. Missing some important bits.
doc---- It’s all in the author’s head… or so zie claims.
Extras: list 2-letter codes in which the documentation is available. E.g. doc++[en,sp,fr]
Extras v2: for each code or codes, separately rate thoroughness. E.g. doc---[en]:+++[jp]
If not specified in extras, the default is English.
E.g. languages: Laadan doc++++; NLF2DWS doc--
Goals: ye generic hobbyist: doc*; ye generic cryptolinguist: doc----; ye generic auxlanger: doc++++
CoRPus: Variety and volume of in-language material available.
crp++++ Thousands of novels, audio/video recordings, poetry, scientific documents, etc.
crp There’s a Babel text…
crp---- You mean I’m supposed to *use* this language?
E.g. languages: English crp++++; ye generic sketch conlang crp--
Goals: ye generic loglanger crp*; ye generic auxlanger crp++++
FINishedness: Stablity & completion of the grammar and vocabulary.
fin++++ Essentially not changing at all.
fin+++ Not changing *intentionally*, but is changing through usual natlang social mechanisms.
fin++ Quasi stable. Still in progress but no major revisions expected.
fin---- Might completely change without warning tomorrow.
E.g. languages: NLF2DWS fin--; Lojban fin++(++++)
Goals: ye generic artlanger fin*; ye generic auxlanger fin++++
FIDelity: How much the map looks like the territory.
fid++++ Everything possible is a near perfect *matopaeic match.
fid++ Most things are at least “translucent”, i.e. they make sense as representations once you know what they mean.
fid---- Everything is completely arbitrary. You would never guess what something means without being told.
E.g. languages: ASL: fid++(+++)
Goals: ye generic non-engelanger: fid*; NLF2DWS: fid++(+++)
EFFort: How hard it is to produce / understand sentences (includes both physical and mental effort). Also includes length of time required.
eff++++ Takes hours to figure out how to say it, and then leaves you exhausted to actually try.
eff---- Very easy and fast to speak and comprehend.
Extra: Separate difficulty from time. (Format: eff[difficulty]:[time])
E.g. languages: Speedwords eff--:----; Ithkuil eff+++(++++)
Goals: ye generic auxlang: eff----; ye generic artlang or engelang: eff*
DeNSity: How much information is transmitted in a certain amount of space (or time).
dns++++ Could write the whole plot of Hamlet on one piece of paper.
dns++ One page of English text translates to 20 seconds of speech.
dns---- One page of English text translates to *20* pages…
E.g. languages: Ithkuil dns++; formal Japanese dns--
Goals: ye generic engelang: dns++(++++); ye generic artlang: dns*
CLaRity: Ease of saying exactly what one wants, no more and no less
clr++++ Can be both completely ambiguous about whatever one wants and retain absolute precision about the rest.
clr About the normal amount of ambiguity or forced specification for natlangs.
clr---- Forced to over-specify or under-specify greatly.
Extra: Distinguish “allowed” from “forced” clarity. (Format: clr[allowed]:[forced] E.g. clr+++:--- = can be very specific, but language requires specifying e.g. everybody’s gender, sexual orientation, and hair color when referring to them.)
E.g. languages: Lojban clr++; ye generic gender-neutral artlang clr:++ (= clr0:2)
Goals: ye generic loglang: clr++++; ye generic artlang clr*
NoiSE resistance: How well utterances are understood when in noisy situations (shouting, bombs, static, bad TV reception, speech impediments, etc)
nse++++ You could use it in the middle of a heavy artillery barrage and still be able to discuss the finer points of Kant as applied to astrophysics without being misunderstood.
nse Fairly hard to get the message right if there’s a lot of noise; might be misunderstood if there’s a mild amount.
nse---- Anything less than perfect clarity means you’re unlikely to get the message even close to correct.
Extra: Semantic difference type, aka the Hungarian phrasbook index (Format: nse[resistance]:[diff type]
nse:++++ Small difference in form à vastly different or completely unrelated meaning (e.g. mother à Chomskian meta-pragmatics or mother à “I cannot permit your ugly face to continue existing, prepare to die”). Aka “tourists likely to get killed”.
nse:---- Small difference in form à similar or less specific meaning (e.g. mother à brother or mother à relative). Aka “graceful degradation”.
E.g. languages: ye generic triconsonantal-root language (for vowels only): nse:--(---)
Goals: ye generic engelang: nse++++:----
Form / Concept Complexity matching: How much complex ideas have complex forms, and simple ideas simple forms.
fcc++++ “Now” = 1 phone. Postnuclearantidisestablishmentarianism = 20.
fcc About even distribution
fcc---- Postnuclearantidisestablishmentarianism = 1 phone. “Male” = 20.
!fcc Random relationship
FAMily: Extent to which this conlang is part of a greater family of languages
fam++++ Part of an extended family of languages, with multiple intersecting lines, intermixing, protolanguages, etc.
fam Part of one line of languages (e.g. “Old Conlish” vs “New Conlish” vs “modern slang Conlish”)
fam---- What family? This language stands on its own.
!fam++ This con-creole is plausible… if Malaysia ever *did* invade Morocco... (Rating = how well it fits in its adopted real-world-natlang family.)
MODalities: Diversity and robustness of language’s modes
mod++++ Can be spoken, written, signed, or maybe even smelt; utilizes each mode’s medium uniquely; each mode conveys full spectrum of meaning (e.g. emotion, undertones, etc)
mod Has a unique writing system
mod---- Only exists in one mode.
Note: Most conlangs are oral with incidental written codes. Some add a more robust writing system, e.g. ideographic. More extensively developed languages may require a separate evaluation for each mode (i.e. treat the oral, sign, and written languages as separate languages in the same family).
DireCTness: How straightforward the language’s idioms are
dct++++ Wanna fuck?
dct Some of both…
dct---- I see a book I might read. Do you have it? *
!dct The language doesn’t have idioms as such, because it isn’t strongly tied to any particular culture.
* from Sally Caves’ Teonaht (“Kyam rykken ellepma wemry hain. Ain fy dhar ha?”)
TiLT: Overall rating of all factors not covered above (Reviews only)
tlt++++ This is the best language I have ever seen.
tlt++ Unusually good.
tlt About average.
tlt---- This is the worst language I have ever seen.
Extra: Separately specify tilt for aesthetics vs technique. (Format: tlt[aesthetics]:[technique].)
E.g. tlt++ = excellent overall. tlt++:- = very pretty, but not particularly technically adept.
Goals: ye generic artlang: tlt+++:*; ye generic engelang: tlt*:+++
CRITERIA - WRAPUP
AMBition of design goals: Difficulty of implementing the author’s design goals
amb++++ Might well require a lifetime’s hard work by a brilliant mind.
amb Would require at least a few months to accomplish.
amb---- A newbie could do that over a weekend.
Note: the amb score should take into account the number of design goals listed, how difficult each one would be, how difficult it would be to fulfill competing goals, etc.
E.g. languages: ??
Goals: ye brilliant artlang: amb++(+++); English relex: amb----
SUCcess: Success at fulfilling the stated goals of the language. (Reviews only)
suc++++ Scores 0 = perfect.
suc Scores 5 = average / needs work.
suc---- Scores 9 = managed to do the exact opposite of what was intended.
!suc Goals are not available.
To calculate the SUC score, for each item – except AMB – for which there is a goal rating available (i.e. not * or unlisted), take the absolute value of the goal rating minus the review rating. Average all these numbers and round down.
E.g. goals: eff4 mlf4 cpx-2 (= eff++++ mlf++++ cpx--)
Review: eff4 mlf-1 cpx0 (= eff++++ mlf- cpx)
SUC: avg(abs(4-4), abs(4-[-1]), abs(-2-0)) = avg(0, 5, 2) = 7/3 = 2.3 = suc++
In all of the foregoing, I have tried to be meticulous in emphasizing that these criteria are completely value-neutral… of themselves. No criterion is inherently more or less ‘important’ or ‘valuable’ than any other.
However, it is nonetheless true that people both have goals for their own languages, and beliefs or values as to what makes for a better or worse language (or at least, one that they personally like more or less). This is reflected in using this framework to declare one’s goals.
I believe the most relevant evaluation, then, is one where many people rate a language on the specific criteria above, and this is compared against the author’s goals to determine how well they met those goals. This score is admittedly subjective in many of its parts, but at least cannot be criticized as applying an inapplicable standard, as it is the author zirself who decides what criteria are important.
This also allows for an easy way to define conlang “schools” as fuzzy categories. Using various statistical and neural net methods – with people desiring similar things generally grouping together – one can distinguish certain broad or specific categories of goal-sets. Keeping with fuzzy categories avoids the too-rigid and perhaps polemic concept of fixed ‘schools’ as such. However, one would expect to see a grouping of e.g. artlangers and loglangers separate out if you only look for two groups, as the two tend to value very different things.
An “ideal” or truly “successful” language is, thus, one that reaches all its design goals perfectly.
Some design goals will be more ambitious than others of course, as reflected in AMB. I think we can agree that a language that is highly successful *and* ambitious is more deserving of praise than one that is only one or the other.
I believe it to be undeniably obvious that conlanging is a *craft*, like being an architect of a living system. All architects must incorporate some degree of both aesthetics and technical mastery, but which aspects are more relevant will depend greatly on the project.
1. Figure out what you are really aiming for – both in the context of the ideas mentioned here, and ones not covered – and publish your goals along with your language. This framework should help, as a starting point, to clarify for yourself what sort of language you are trying to create. Invite people to review your language.
2. Review both your own conlangs and others’. Use this framework as a beginning from which to review a language – to give the review from structure and “get to the point” with ratings. Supplement it with more detailed descriptions for those points that are interesting, need work, or are done particularly well.
Please note, this system intentionally does *not* cover linguistic typology or linguistic features. There are simply far too many to cover; doing so would require another, rather more cumbersome, system. More importantly, though, linguistic features are not *inherently* purposeful, and only indirectly related to the ‘feel’ of a language or to whether one is likely to think more highly of it. This system is meant to rate those more core, meta elements of how a language is perceived.
I have tried to ensure that this system is maximally comprehensive for 95% of people, and has a minimal amount of overlap between its criteria. It is not intended to be able to be 100% comprehensive; that is not possible in any finite rating system. It should however cover the goals of all types of languages; most languages will probably find many of these ratings irrelevant, and that is fine. The point is, different languages will find different things to be relevant.
However, I may have overlooked something; if so, please let me know, and we can make version 2.1. ;-)
[1, 2] these entries are archived, with retrospective annotation by Jesse Bangs, here: http://web.archive.org/web/20041022065450/http://students.washington.edu/jaspax/conlang.htm
This can be used in conjunction with the conlang code v1.1 above. To do so, use the format:
[conlang code v1.1 or later minus ‘name of conlang’ item]
CLCv2 [date] G language(s)[priority] [insert CLCv2 code here] [linebreak]
G language(s)[priority] [insert CLCv2 code here] [etc as relevant]