Virtual Dhamma-Vinaya Vihara

Studies, projects & library - [Studium, Projekte & Bibliothek] (brahma & nimmanarati deva) => Translation projects - [Übersetzungsprojekte] => Studygroups & Dhamma Dana - [Studiengruppen & Dhamma Dana] => Zugang zur Einsicht - [Access to Insight] => Topic started by: Dhammañāṇa on February 11, 2019, 01:08:33 PM

Title: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on February 11, 2019, 01:08:33 PM
Writing in Khmer without the use of zero white spaces is not very good, yet less editors and programs would allow to show them.

Are there possibilities to let them be shown in ati-editor and on the compare- page?

A topic was raised in DW-Forum as well: Display of "zero width space" and other unicode characters (https://forum.dokuwiki.org/thread/16692)
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 02, 2019, 09:37:10 AM
Just tried and added a button to the editor toolbar to switch on displaying zero-white-space characters. (See screenshot attached for how it look on a big desktop screen.)

It is a very hacky solution, and currently only working on a large screen. There is no way to directly render zero-width characters in HTML, so one has to indirectly copy/calculate the text layout and display marks in the right spots, which is very slow for large texts with many zero-whitespaces.

Tested for http://accesstoinsight.eu/km/tipitaka/book_053?do=edit on my desktop computer, it takes at least fifteen seconds here, after clicking the new button, before the browser becomes responsive again shows the blue marks. And when changing the text, one has to wait again, because the display of the white-space marks has to be calculated for the whole text with every change.

For very small texts it seems to work conveniently on my computer. But after some ten thousand characters, and especially with many zero whitespaces it becomes very slow quickly.

On Android smartphone (and probably any smaller screen), the layout is currently not usable at all, showing everything completely wrong.

Maybe possible to find some optimizations with time.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 02, 2019, 10:16:01 AM
Sadhu

(works as told and runs strange if zooming)

Atma came "CSS @font-face Rule", with which certain unicode-range can be addressed, to mind. Not sure if such as background or styling can be addressed with it.

Quote from: https://www.w3schools.com/cssref/css3_pr_font-face_rule.asp
   unicode-range    Optional. Defines the range of unicode characters the font supports. Default value is "U+0-10FFFF"

An idea came here to mind: what if there is a 'special' "white space font" wherein the zero width space is a slight box with a small point.

The "tool" then "simply" switches to another font which has the character not with a zero width and possible a small sign to recognize.

Or even generally using a font that is available at ati.eu server for the editor, editor friendly and Khmer-font friendly (incl. size, distances of lines... for certain character ranges).
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 03, 2019, 01:59:53 AM
An idea came here to mind: what if there is a 'special' "white space font" wherein the zero width space is a slight box with a small point.

The "tool" then "simply" switches to another font which has the character not with a zero width and possible a small sign to recognize.

That sounds like the most straightforward way if something like that is possible.
Not familiar with editing fonts and how they are actually stored, somehow I have doubts that it could be so clean and "easy", I will try to find out what is possible and impossible. Sadhu!

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 03, 2019, 06:34:09 AM
They are often ttf-files, usually they are stored anywhere on certain server but can also stored on the own (at least if allowed by the creater, Atma did such for the Pali-Khmer script, called by "div cs-km"). There are certain programs, like grafic programs where one can desighn each character. There is of cource certain new knowledge needed and a punsh of "new" words and funcutions in detail but possible not that huge for simply make enlarge the zws to a small space and draw a point. Yet a given front is needed which also covers all scripts.

Maybe Nyom Nathan ( sungkhum ) , much infolved in the matter, knowing also designer) feels inspired to help that a script might be remorseless useable for the Sangha and has possible friends helping also with a "zero width space font". Not sure how libre office, for excample, managed the display.

Years ago my person "loved" drawing and scripts design, but it's truly not really proper anymore to engage much here but let others do merits instead. Surely a wonderful area for such, given that the Sangha has actually not even scripts to use, given.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 04, 2019, 12:07:06 AM
Sadhu. I was aware of the various file extensions, but not how they are stored inside, and especially if it is really possible to make visible glyphs for invisible space or even other "control characters" (like line break etc.).

After some searching and trying, although no definite answer (https://graphicdesign.stackexchange.com/questions/51233/is-there-any-font-where-white-space-or-non-printing-characters-are-visible-for), I found that it is apparently impossible to just insert a glyph for zero whitespace that would then show up in HTML or text editors:
Quote from: comment by Wrzlprmft on graphicdesign.stackexchange.com (https://graphicdesign.stackexchange.com/questions/51233/is-there-any-font-where-white-space-or-non-printing-characters-are-visible-for?newreg=7a166240ce7d453e99d007530158e814#comment-73708)
Note that many programs manage their whitespace themselves and, e.g., only use the width of the font’s space character. ...
With font editors, such as FontForge, one can create and assign such glyphs to any code point, including zero white-space (u200b) etc. But they are then not displayed in web browsers, or most editors, because most softwares just have their standard way of treating these special invisible characters without even looking if a glyph is defined in the font for them. Tried to make a glyph for zero white-space but it is simply not rendered.

However, as a workaround, one can just use another Unicode code point (from the Unicode "Private Use Area" (https://de.wikipedia.org/wiki/Private_Use_Area) which are not assigned to anything by the standard) instead of zero-whitespace, for which one can then define a glyph, and simply replace all zero-whitespace with that other code point during editing, and replace it back before saving. This would still require some JavaScript, but much simpler with no complicated calculation involved so that it would not slow down much or display garbage.

Some changes to the DokuWiki inner PHP code are needed, which is structured a bit complicated and confusing to me. So it may take some time to figure all out that is necessary.
At some point this evening I may be editing the necessary code on the server, so editing things may look strange or not work for a short time.
/me Not really capable doing this anymore for today.

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 04, 2019, 04:34:24 AM
Understandable, also that zwsp has been treated alien in programming.

Not sure if such helps if building on an editor like https://www.dokuwiki.org/plugin:codemirror
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 05, 2019, 12:23:25 PM
Sadhu, yes. Probably best to do it as a plugin, for which the CodeMirror plugin is useful as an example, instead of butchering internal DokuWiki code.

Still needs some time to figure things out.
_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 05, 2019, 01:34:09 PM
Just to let it be know (Atma thinks that it's know, but to do not forget at time then): the css wiki... might "confuse" the z-position, not only making a gray area on the left of the box but also getting the scroll area above the edit and menu section right of the text.

Atma made Nyoms effort and undertaking a little bit known: A "showing invisible character editor plugin" (https://forum.dokuwiki.org/thread/16883)

Mr Myron has developed (https://mturner.org/devel/vkeyboard) a key board plugin, which (others then the most mobile keyb can type zwsp and has all characters included) does not require much issues for the writer online then as well: https://www.dokuwiki.org/plugin:vkeyboard if thinking that such is useful.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 18, 2019, 04:34:55 AM
_/\_ _/\_ _/\_



The zero white-space plugin should now be working (all files contained in lib/plugins/zws), displaying zero whitespace in the text editing area and on the revision comparison page:
(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7469)

There is a button to insert a zws at the current position in the text:
(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7467)
Otherwise no changes in user interface.


The way it works is like this:

There is a font (lib/plugins/zws/zws2.ttf) with only one character, for the unicode code point E000 (written as '\ue000' in JavaScript and PHP), because a character for the actual ZWS code point 200B ('\u200b') would not be displayed in browsers.

This font is referenced in lib/plugins/zws/zws.css and applied only to the text editing area and the revision comparison table. (It does not matter that the font has only a symbol for one character. All other characters will be displayed by the next fonts on the "chain".)

During editing of a page, all ZWS characters are replaced by the placeholder E000, which is displayed by the font. When saving, the E000 character is replaced again by 200B (normal ZWS).


Very simple on principle, but some tricks were needed, because browser APIs are sometimes complicated when dealing with unusual cases.
So I hope that everything really works now. Feedback is of course very welcome.

 *sgift*




Some possible improvements:

Maybe the ZWS placeholder could be thicker? (Would be quick and easy.) Maybe colored? (Not sure how to make fonts that are colored by default. I think it is possible somehow.)

Maybe a button to turn ZWS display on and off? For now, it is always turned on (in the editor). Maybe somewhere on the normal page UI, a switch to use the ZWS font everywhere when reading would be good, but might need some tricks again.

_/\_ _/\_ _/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 18, 2019, 06:31:50 AM
Sadhu, Sadhu
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 18, 2019, 05:09:52 PM
My person did not experiment much till now (thought of how would it insert copied text including zwsp) but came across that when copy form one editor window to another it seems that it suppresses line breaks (better replace with a white space).

Copy:
(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7471;image)

past:
(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7473;image)

Copy it here, it seems to come from the source and does not happen in the moment of past:

Code: [Select]
==== 1. Kimilasuttaṃ ==== <span sang_id #{file-}.{no}>[[{path-release}:{file-}.{no}|{file-}.{no}]] | [[{path-source}:{file}#{file-}.{no}|source]]</span>
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 18, 2019, 08:58:30 PM
I see. I think it should be fixed now. (Maybe needing to clear cache by adding "&purge=true" to the URL, possibly also helpful to open and refresh the CSS and JS files http://accesstoinsight.eu/lib/plugins/zws/zws.css (http://accesstoinsight.eu/lib/plugins/zws/zws.css) and http://accesstoinsight.eu/lib/plugins/zws/zws.js (http://accesstoinsight.eu/lib/plugins/zws/zws.js) so that they are refreshed in browser cache).

I had wanted to automatically replace the placeholder with ZWS when copying from the editor. So that it would be a real ZWS when pasting somewhere else again. But it seems that this modification also removed the line breaks and I see no way to make it work like intended. So it has to work without this convenience for now.

On the other hand, when inserting text into the editor, ZWS characters will be automatically replaced with the visible placeholder, and this should still work as intended. (But it is not possible to do "undo" after such a paste, if there were zero-width characters in it that were replaced.)

_/\_

/me hoping that it now really works after hudling to fix it, might not be much available in the next days

/me Edit: Oops, something again not working right: Now ZWS is not replaced in the text loaded from the server. Hope I can quickly make it work. Should work now.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 19, 2019, 10:32:21 AM
Sadhu

The first test showed that text copied into the editor would be rendered in regard of certain html characters and what ever content is in the editor, can not be copied anywhere else.

http://accesstoinsight.eu/user/johann/test

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7476;image)


The second test, after having called the posted links, shows that the renderings of special characters also become part of the displayed text.

http://accesstoinsight.eu/user/johann/test1

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7478;image)

Atma is a little bit "scared" to open any page for now. Which seems to be good, since it obviously converts the text, having downloaded the file after saving:

Code: [Select]
==== សុន្ទរសមុទ្ទត្ថេរ ====

&lt;span sang_id #sut.kn.tha.07.01&gt;[[km:tipitaka:sut:kn:tha:sut.kn.tha.07.01|sut.kn.tha.07.01]] | [[km:tipitaka:book_057#sut.kn.tha.07.01|book_057]]&lt;/span&gt;



&lt;div centeralign&gt;**(១. សុន្ទរសមុទ្ទត្ថេរគាថា)**&lt;/div&gt;



&lt;span para #para_225&gt;[២២៥]&lt;/span&gt; ស្រីផ្កាមាស​តាក់​តែង ស្លៀក​ពាក់​ល្អ ទ្រទ្រង់​កម្រងផ្កា ស្អិតស្អាង​ហើយ មាន​ជើង​ស្រឡាប​ដោយ​ទឹក​ល័ខ ពាក់​ស្បែក​ជើង លុះ​ដោះ​ស្បែក​ជើង​ចេញ​ហើយ ក៏​ធ្វើ​អញ្ជលី​ចំពោះ​មុខ ស្រីផ្កាមាស​នោះ និយាយ​នូវ​ពាក្យ​ជា​ប្រធាន​នៃ​មិត្ត((ពាក្យ​ចែចង់។)) នឹង​ខ្ញុំ ដោយ​សំដី​ដ៏​ទន់​ពីរោះ​ថា លោក​នៅ​កំឡោះ​ហើយ​បួស លោក​ចូរ​ឋិត​នៅ​ក្នុង​ពាក្យ​ប្រដៅ​របស់​ខ្ញុំ លោក​ចូរ​បរិភោគ​កាម​ជា​របស់​មនុស្ស​ចុះ ខ្ញុំ​នឹង​ឲ្យ​សម្បត្តិ​ដ៏​ពេញ​ចិត្ត​ដល់​លោក។ ខ្ញុំ​បេ្តជ្ញា​ពាក្យ​ពិត​ដល់​លោក ឬ​នឹង​នាំ​ភ្លើង​ដល់​លោក។((ស្បថ​នឹង​ភ្លើង)) កាល​ណា​យើង​ទាំង​ពីរ​នាក់ ជា​មនុស្សចាស់​កាន់​ឈើច្រត់​ហើយ យើង​នឹង​បួស​ទាំង​ពីរ​នាក់ នឹង​កាន់​យក​នូវ​ជ័យជំនះ​ក្នុង​លោក​ទាំង​ពីរ។ ខ្ញុំ​បាន​ឃើញ​ស្រីផ្កាមាស​នោះ មក​ធ្វើ​អញ្ជលី តាក់​តែង ស្លៀក​ពាក់​ល្អ ដូច​ជា​ជាប់​អន្ទាក់​មច្ចុ លំដាប់​នុ៎ះ ខ្ញុំ​ក៏​បាន​ធ្វើ​ទុក​ក្នុង​ចិត្ត​ដោយ​ឧបាយ​​នៃ​បញ្ញា។បេ។((គប្បី​មើល​ក្នុង​ចតុក្កនិបាត។))

&lt;div rightalign&gt;សុន្ទរសមុទ្ទត្ថេរ។&lt;/div&gt;



==== លកុណ្តកត្ថេរ ====

&lt;span sang_id #sut.kn.tha.07.02&gt;[[km:tipitaka:sut:kn:tha:sut.kn.tha.07.02|sut.kn.tha.07.02]] | [[km:tipitaka:book_057#sut.kn.tha.07.02|book_057]]&lt;/span&gt;


It also seems to convert into CRLF line breaks.

Don't open any page in the editor and save the content for now!

It seems that the script also adds zwsp next to the html-value of certain characters.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7480;image)

If having less time and much in hurry, maybe it's good and possible to disable the modification meanwhile.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 19, 2019, 11:41:21 AM
Vandami Bhante _/\_

That comes from hudling.

/me Oh... I just notice, when clicking the "<.I.>" smiley here in the forum post editor, it inserts "&lt;.I.&gt;", instead of <.I.>. Must be a similar problem...   ^-^

Now I am confident everything should really be fixed. Sorry for any troubles. But at least nothing could have been lost, with old versions being kept in revision history.


Problem 1: inserting &gt; instead of > etc. happened during loading of the text for editing (in /lib/plugins/zws/action.php on line 86 at the time of this writing, applying formText function to the text, which DokuWiki applies afterwards another time, so it was done twice [converting into &lt; etc. for display is necessary but should happen only one time]). Is now fixed.

Problem 2: Not being able to copy from the editor, was leftover from the previous attempt of using contenteditable instead of textarea, using a method that does not work with textarea (in zws.js, lines 64-72 at the time of this writing). Is now fixed (only using standard copy now without modifications).

Problem 3:
Quote
It also seems to convert into CRLF line breaks.
Must have been the same cause as the first problem.

Problem 4:
Quote
It seems that the script also adds zwsp next to the html-value of certain characters.
Must have been the same cause as the first problem.


All this should now work. I don't think it is necessary to disable modification. Old versions will remain in history and can be restored in case that really still something is wrong. And nothing really dangerous would happen.

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 19, 2019, 12:52:13 PM
Sadhu, Sadhu... may my person never hudles with comments and tests.  :) May Nyom spend a joyful Uposatha!

Atma will test on a little.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 19, 2019, 01:02:27 PM
The previous html-value things are gone, copy and paste works so far with a special when copy from the editor window to libre-office (for example). Actually looking nice but maybe not intended:

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7482;image)

The comparison view seems to add "fictive" zwsp, but looks great as well so far.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7484;image)

Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 19, 2019, 11:39:50 PM
The previous html-value things are gone, copy and paste works so far with a special when copy from the editor window to libre-office (for example). Actually looking nice but maybe not intended:

No, not intended. But understandable. The unicode character E000 is used instead of ZWS to display the font. And that is also what is copied now, when copying from the textarea (since modifying the copying content did not work: removing line-breaks when trying to do so).

Since E000 belongs to the Unicode "Private use area" (https://en.wikipedia.org/wiki/Private_Use_Area), its look is not defined by the unicode standard, but different fonts can define it as they want for their purposes.

I had expected that E000 and other "Private use area" unicode characters would not have a symbol defined in most fonts, displaying a generic placeholder sign (like an empty square, or a square with the code 'E000' written in it) instead.

So I thought it would just be displayed normally as just a placeholder square, something like this:
Code: [Select]
|E0|
|00|
(displayed as one small square symbol).

But it seems that for example LibreOffice will always try to find a font which has a character defined for this symbol, and use it. In this case there must be some font installed on the system where the character is defined and LibreOffice is automatically using that to display the E000 character. Otherwise it would be displayed as just a placeholder.

In any case, it is not possible for now to automatically replace the character with a real ZWS when copying. So one has to take care to replace it when editing in some other program. (But actually, pasting these characters back into the text area in ATI DokuWiki editor will not be a problem. It will then be saved as ZWS in DokuWiki.)

The comparison view seems to add "fictive" zwsp, but looks great as well so far.
Ah, now I think I know where this would come from. I have seen some function called "insertSoftBreaks" or similar, defined somewhere in the DokuWiki PHP code, and this is called when rendering the revision comparison page.
DokuWiki would insert ZWS characters on the comparison page, so that long code sections without space would be broken into different lines at some fitting positions, and not displayed as one huge line reaching out of the screen.
So I should try to make sure that the ZWS replacement happens before the insertSoftBreaks when displaying the comparison page. Should be possible to do so in the next days.

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 20, 2019, 06:31:30 AM
Sadhu

If working further on a document outside of the editor one needs to replace the character then. Just a thought: what if the whitespace view (possible displaying linebreak, ws, tabs as well, similar like libre or word) can be turned on and off in the editor (where when off the normal character appear).

Amazing the effort here.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 20, 2019, 10:26:20 AM
I think the best solution might be to simply use the already existing and previously mentioned (http://forum.sangham.net/index.php/topic,9136.msg18505.html#msg18505) CodeMirror plugin (https://www.dokuwiki.org/plugin:codemirror).

It has all the desired features for the editor, including display of ZWS, whitespace and line breaks, possible to turn on and off. I had previously not tested it and thought that ZWS display would not be included. But it is, and it all seems to be working perfectly.

ZWS is always displayed as red dots, taking up one space, and not possible to turn off. Perhaps I can find a way to toggle them on/off as well.


The only additional thing remaining from my plugin attempt would then be the display of ZWS on the revision comparison page.

For now, I have disabled the ZWS plugin and installed CodeMirror instead.

For the ZWS display in the comparison page to work properly (without inserting additional ZWS between tags etc.), it would be necessary to change something in the DokuWiki code directly, which is not possible with a plugin.
And the little bit of JavaScript and CSS can be included in conf/userall.css and conf/userscript.js, so no real plugin needed anymore for that.

Edit: Comparison page should be working now, only showing the ZWS where they really are. All necessary for that is in conf/userall.css (marked with comment), conf/userscript.js and line 1366 of inc/html.php, which was changed from
Code: [Select]
        echo html_insert_softbreaks($diffformatter->format($diff)); ?>
to
Code: [Select]
        echo html_insert_softbreaks(str_replace("\u{200B}", "\u{E000}", $diffformatter->format($diff))); ?>

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 20, 2019, 11:38:13 AM
Sadhu, Sadhu

Didn't came across that this mod shows zwsp and great that the display of zwsp in compare view works. The possibility to write a zwsp might be great since certain keybords often lack of such.

Not sure if a "way to toggle them on/off" is needed. When copy to word or odt, all seems fine. Sure, the display is somehow "brutal" and confusing, giving ws, tabs or linebreaks not much sign.

Great work and much place to take ones goodness along, rejoice with it, and possible go on another time once again, delight in giving into something for good and liberating purpose. Mudita!
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 20, 2019, 12:30:15 PM
Sadhu, Sadhu

Didn't came across that this mod shows zwsp and great that the display of zwsp in compare view works. The possibility to write a zwsp might be great since certain keybords often lack of such.
Ah yes, the button. I will try ti add it again.
There was also the vkeyboard plugin mentioned before. I had tried it out a bit, but does not seem to work. Not updated for almost five years, it might be that some things have changed and become incompatible. But it is surely worth a look to see if one can find good methods there to make something like this work here again.

Not sure if a "way to toggle them on/off" is needed. When copy to word or odt, all seems fine. Sure, the display is somehow "brutal" and confusing, giving ws, tabs or linebreaks not much sign.
I don't understand the last half-sentence "giving ws, tabs or linebreaks not much sign".
The red zero-mark looks "brutal", yes, maybe possible to change that into a thin line instead of taking up a complete space. But normal white-spaces/tabs/linebreaks might be better shown more strongly visible? Or how was it meant?

Great work and much place to take ones goodness along, rejoice with it, and possible go on another time once again, delight in giving into something for good and liberating purpose. Mudita!

Good to hear that it appears to be useful so far.  :)

_/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 20, 2019, 01:10:58 PM
Quote from: Upasaka Moritz
I don't understand the last half-sentence "giving ws, tabs or linebreaks not much sign".
The red zero-mark looks "brutal", yes, maybe possible to change that into a thin line instead of taking up a complete space. But normal white-spaces/tabs/linebreaks might be better shown more strongly visible? Or how was it meant?

Editors, at least if "more professional" usually show slight signs for certain line breaks, tabs, spaces (also word has this, this pi-like icon to switch on and off) and since syntax works much with them possible fine to have decent marks and zwsp is just one under them (red and brutal possible because it's a programmers enemy character, as far as it can be read and seldom used as important part of text). LibreO. uses decent signs. But it's all merely already comfort and for working it's great as it is as well. Surely the red point is very dominant and leaves one to oversee normal spaces.

Quote from: Upasaka Moritz
Good to hear that it appears to be useful so far.  :)

_/\_

Act-ually the quality of mind and it's object (joy with the object of ones scarifies), it's pureness counts and that is something only attained and gained by the doer and particularly shared by letting others known ones deeds. It's usual that such also is of much use in merely wordily manners as well, pleasing and easy nourishment. Mudita.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 26, 2019, 08:24:08 PM
Just found out that there are even a lot of settings incl. the display of white spaces and line breaks. The layout and frontsize options are also very useful. Althought disable spell check, it's also possible to disable the editor to normal.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7486;image)

In regard of the "styling" of the zwsp Atma investigated a little the origin sphere of the editor and asked the developer, Mr. Marijn (https://discuss.codemirror.net/t/zero-width-spaces-and-styling-of-white-spaces/2054), so to possible find out how and where to simple edit.

Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 27, 2019, 10:42:43 AM
Just found out that there are even a lot of settings incl. the display of white spaces and line breaks. The layout and frontsize options are also very useful. Althought disable spell check, it's also possible to disable the editor to normal.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7486;image)

In regard of the "styling" of the zwsp Atma investigated a little the origin sphere of the editor and asked the developer, Mr. Marijn (https://discuss.codemirror.net/t/zero-width-spaces-and-styling-of-white-spaces/2054), so to possible find out how and where to simple edit.

Thanks to the hint I was able to make the zws a bit better looking (see screenshot). Now in light blue (for now not possible to turn on/off, will try to add option soon).
Other special characters (from the range of special characters defined by CodeMirror (https://codemirror.net/doc/manual.html#option_specialChars)) are still rendered as red dots.
If there are any suggestions for better colors, also for the space and line-break markers (shown when enabling the options "Show Whitespace" and "Display line numbers", respectively), I could change that as well.

_/\_

/me The "line break" marker seems a bit buggy when using "Highlight current line" but not "Show line numbers". Then the marker will appear only at the end of the current line and stay there even when selecting a different line. Might be confusing. I will try to find out the reason. Maybe would be good to just combine "Highlight current line" and "Show line numbers" into one option, so that the error would never appear.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 27, 2019, 11:11:53 AM
Sadhu, Sadhu

Looks more/used easy for the eye, the color as well, either dark or white.

Maybe the other signs like the point of white space, the line break, can be made in the same color (then its easy to see them as a "kind").

If the zwps could be moved sight to the right, so that it comes between the characters without (like now) changing the width, and if it would be slight broader, then it might be even more easy to discriminate required and not so different as used.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7488;image)

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7490;image)

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7492;image)

Not sure if it is possible to use also a sign, like an arrow, for tabs.

Btw., by clicking Ctrl+F one can search within the text, also with regex, and all finds would be marked.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on April 27, 2019, 10:42:07 PM
Ah, I see. That must be because of using different fonts.

On my system, CodeMirror was using the default monospace font for the text area, while on the pictures shown above, a different (non-monospace) font is used (which is probably not installed on my system, and therefore the monospace font used instead).

I have inserted the ZWS marker as a simple '|' symbol, shifted slightly to the left, and overlayed, so not taking up any space of the actual text behind. The positioning would be different for different fonts. I chose -0.25rem (rem is a unit of text size in CSS, 1rem = current line height or something), which should be more or less correct for most monospace fonts.

I think between non-monospace fonts, there may be more differences, so it might not be possible to find a left shift value that works perfectly for every font.

For now, I have changed the CSS to use the borrowed "Moul Pali" font in the editor, and set the zws left shift to -0.125rem, which seems to fit well together. But I am not sure if Moul Pali is really a good font choice.

Since the Moul Pali font was actually not really used before, the Khmer texts seen on accesstoinsight.eu were always in some "default" fonts. So it seems there should not be a necessity to specifically include a font to support Khmer script. Khmer script seems to be included in most or many "standard" fonts nowadays, and browsers would automatically choose a font where the Khmer character is defined in most normal cases (if not explicitly setting a CSS rule to use _only_ one specific font, without default "fallback", which has no Khmer characters).


I also changed references to "Moul Pali" in conf/userall.css, because they were all written wrong. (Need to be put into quotes, because of space in the name.) So "Moul Pali" was probably actually never used before.

Not sure where the following CSS rules are now visibly applied. Maybe Bhante might check if they are actually looking as intended:
Code: [Select]
/*pali khmer font span*/

@font-face { font-family: "Moul Pali"; src: url('Moul Pali.ttf'); }


.dokuwiki div.wrap_pali_km {
font-family:"Moul Pali", sans-serif;
font-size:80%;
}

.dokuwiki span.wrap_pali_km {
font-family:"Moul Pali", sans-serif;
font-size:80%;
}

/* ... */

.dokuwiki div.wrap_cs-km { font-family:"Moul Pali", sans-serif !important;
}


I set ", sans-serif" everywhere behind as a "fallback" (although Moul Pali is more of a Serif font) in all those cases where using "Moul Pali" as font. This means that if a character is not defined in Moul Pali, it will use the standard of the "sans-serif" family instead. This is how it generally works in CSS: if a symbol is not found, the browser will look in the next font of the list, and in general it would always fall back on one of the three "standard font families": monospace, serif or sans-serif. Normally, Moul Pali would fall back on "serif" for non-Khmer symbols, using whatever is the standard serif font on the system. But sans-serif seems generally better to read and fitting with the general DokuWiki style.


Maybe it would be better to not use "Moul Pali", because it is more like a Serif font, and some things can look a bit strange (see screenshot (http://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7494;image)).
(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7494;image)

I think it might be best to use the standard font that is used on DokuWiki pages, which is "Arial, sans-serif" (i.e. using Windows "Arial" font if it is available, or otherwise whatever standard "sans-serif" is installed on the system).

For now, the editor font is set as "Moul Pali, sans-serif". But there is also a button now to choose a different font (need to enter a correct font name that is recognized on the system), so one can try out different fonts if one knows some font names (however it is probably best for compatibility on different systems to rely mostly on just the standard font-families "serif, sans-serif and monospace", if not having font files available to share).

The "insert ZWS" button is also included again:

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7496;image)
[small]("insert ZWS" and "choose editor font" buttons)[/small]

Adding tab arrows is surely possible, but maybe difficult to implement well with non-monospace fonts and might need some time.


Ah, and I recognized the use of different "themes" in CodeMirror that one can choose. So I thought it is best to use one of the theme colors also for zero-whitespace (instead of a fixed color, that might not fit well with some themes).
The color themes of CodeMirror have a number of CSS classes for certain elements in the text, for example:
span.cm-keyword, span.cm-atom, span.cm-number, span.cm-def, span.cm-variable, span.cm-variable-2, ..., span.cm-operator, span.cm-property, span.cm-comment, ...
, which have different colors for different themes and are used differently for whatever kind of "programming language" CodeMirror is set to use. (DokuWiki markup has its own CodeMirror definitions in the plugin).
For now, I chose the CSS class cm-comment for the ZWS markers. I think the color is good for most themes. But feedback is welcome. One could try different things.

_/\_

Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on April 28, 2019, 02:30:13 AM
Sadhu, Sadhu.

Moul Pali is only a "style/decor" font for Pali-words, not so known and possible had to read since old script. Like old German script after 2 Generations can no more be read. Atma used this sript for the cs-km pages and for the use of pali-words in dictionary as span. For editor general probably not good (my person still has a very hard to read, remember the different characters). It would also require much styling like line spaces 200% etc. since overlapping otherwise like Nyom has already seen.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7502;image)

The fall back fonts on pc and android are different, but always easy to read. Most easy the android used. Fall backs of "Sans" but which is owned by google http://www.khmerfonts.info/fontinfo.php?font=2522

"Koh Santepheap" looks also fine and a nice name "Island of peace"...

PC (windows) uses Khmer OS System which is not so pleasing on small devices but possible best aside of Sans. As there is no font owned by the Sangha, fall back seems to be fine for general script and the tool to select wished scripts is more then comfortable in addition.

Mr Nathan ( sungkhum )might know much and more in regard of scripts, having been involved much in their development.

A screen-shot from mobil device (wsp are here dark-green). Position look perfect.

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7498;image)

(https://forum.sangham.net/index.php?action=dlattach;topic=9136.0;attach=7500;image)

 Changing themes seems to cause different colors for wsp and it's endless and sisiphus-work to look for perfect style. If it looks ok for Nyom on pc and in standard theme, thats great. If one wishes other themes, that has already lot of ways and may be open for inspired gifts all the time.

For working direct in the editor, use dw as editor, the conditions are, aside of the spell-check, really great and comfortable.
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Moritz on March 07, 2020, 01:17:26 PM
Moul Pali is only a "style/decor" font for Pali-words, not so known and possible had to read since old script. Like old German script after 2 Generations can no more be read. Atma used this sript for the cs-km pages and for the use of pali-words in dictionary as span. For editor general probably not good (my person still has a very hard to read, remember the different characters). It would also require much styling like line spaces 200% etc. since overlapping otherwise like Nyom has already seen.

I removed the CSS to use "Moul Pali" as standard script for Khmer in CodeMirror, so now the browser's/operating system's standard sans serif font would be used instead, which should be more readable.

Since Bhante mentioned, I also tested to use "KhmerOS Content" font for it. But I think it looks worse than the system's sans-serif default here in the editor, at least on my computer. Although I am not used to read Khmer fonts, still not really knowing the writing system, but to me the normal system font seems more readable, using more space.
I think most modern operating systems have readable Khmer fonts by default nowadays. So to add additional KhmerOS fonts might be obsolete at least for standard (not "decorative") text.

To compare how the difference looks here (on my computer), see the attached screenshots.

At http://sangham.net/test/fonts.html (http://sangham.net/test/fonts.html) one can also see the difference between one's own browser's/system's standard sans-serif font and KhmerOS Content font. (So the first, "Standard Sans Serif", always depends on the system.)

Which one is more readable for you?



_/\_ _/\_ _/\_
Title: Re: [ati] Possibility to display of "zero width space" and other unicode characters
Post by: Dhammañāṇa on March 07, 2020, 01:32:21 PM
Quite better, the standard Sans-Serif, at least to read.

Nyom Cheav Villa once mentioned to be not able to work in/with the editor because it uses to jump (just because here in this topic) and Atma also came across that it is often hard to mark areas.