autocomplete - Best way to represent language tokens for an autocompletion scenario -

as of know, i'm developing own ide. might think "oh no, one?!" - don't worry, no one's forcing use it, , doubt published anyway.

so, onwards main issue. i'm trying implement autocompletion system. exact ui not concern. however, storing language/library tokens in flexible way, main problem.

let's we're suggesting css selectors or attributes user. we'd have like:

- css/core   -                      // anchor tag   - etc                    // valid html tags   - .stuff                 // class name parsed user project   - ?etc                   // more stuff parsed user project (ids, classes...) - css/properties   - border                 // regular css properties - need associate                            // <border-style> , <color> value tokens   - etc                    // rest of them - css/values/border-style  // property value tokens   - solid   - dotted - css/values/color   - red   - green   - fucshia 

so each token gets namespace can track between tokens. bnf, token values made of subtokens such case border , color.

1. don't forget need store might relate languages exotic syntax. 2. also, important note need somehow merge above information context-dependent one, such list of class names gathered project's files. should fast , efficient, without causing duplicate tokens etc.

so, conclude, thing here complicated, , can't think of way general , flexible solution. keep in mind ide should cater kind of language, making more complicated.

i'm not sure if question better suited in, example, programmers, i'll leave mods decide.

i worked on ide called sharpdevelop. let me start more general discussion before storage question.

i don't think can solve autocompletion in generic way. ides support various languages having plugin each of languages , entirely plugin figure out completion list should based on current position of cursor in document.

the ide provides simple interface plugins implement. example, code in ide showing autocompletion this:

getautocompletionlist(editor) {   plugin = editor.languageplugin;   plugin.getautocompletionlist(editor.cursorposition, editor.parseddocument); } 

a csslanguageplugin , phplanguageplugin have separate implementations of getautocompletionlist - 1 used when editing css, other when editing php.

as others pointed out, context around cursor important. example, when editing following css:

h1 {     text-align: <cursor> 

the contexts be:

[csstoplevelcontext] {     [csspropertycontext]: [csspropertyvaluecontext] } 

the implementation of css plugin following:

// csslanguagebinding getautocompletionlist(cursorposition, document) {     completioncontext = this.getcompletioncontext(cursorposition, document);     // completioncontext {      //     'name': 'csspropertyvaluecontext',      //     'propertyname': 'text-align'      // }     return this.completiondatabase.getcompletionlist(completioncontext);     // returns ['left', 'center', 'right']; } 

now question - completion database. again, (and should) different implementation different language plugins - in php work classes, methods , variables, , have care visibility (private, public, protected). in css work tags, classes , properties.

as correctly pointed out, completion database should consist of:

• common tokens
• tokens imported current project
• tokens in current project itself

in sharpdevelop, 'common tokens' part not there, project imports standard library, enough analyze imported libraries when opening project.

in php same , cache token database seen libraries.

now storage format. offer autocompletion in php, need know current class, base class , interface hierarchy, methods in base classes , interfaces , visibility, variables visible in current context , types (not possible in php) , on.

for reason, think relational database not choice. how store classes, interfaces , methods there , navigate inheritance hierarchy? sharpdevelop stores in memory object model (class has base type, list of interfaces, list of members etc.). 8000 items not large number, , if stored 8000 items in relational database, small database engine keep in ram anyway.

sharpdevelop keeps completion information in memory , when open 700k line project in sharpdevelop, memory consumption still pretty low. suggest initialize autocompletion data structures upon opening of project , keep them in memory. others said, have update them in background user typing (introducing new methods, renaming fields, etc.).

so that's php. css, data structure similar outlined in question seems reasonable. load memory structured file upon start of ide / opening project / opening first css file.

---

as end note, implementing autocompletion css shouldn't hard. php, more difficult , start simple - offering 8000 tokens standard library plus offering words user typed somewhere else in project. such approach used editors sublime text , works surprisingly well.