display utf-8 in browser and jwm: solved?

[orineu]

So I'm cleaning out my bookmarks and went to view a Japanese website, and none of the characters displayed! -fossapup64 9.5, both Chrome and Pale Moon, and also Geany

EDIT: this fix worked, but not with 100% customizability

I think this is what wound up working:

1. install a charset-supporting font using the JWMdesk Font Manager (Migu, MigMix, or M+, for instance) or PPM (Noto-cjk, for instance)
2. restart the entire system (not just the graphical server/jwm)
3. In all jwm font-controlling files, specify the exact name of that font (as displayed in gfontsel, may need to resize the window) followed by the size as below:

Code: Select all

	<Font>Migu 1P-12</Font>
	<Font>MigMix 1P-12</Font>
	<Font>M+1P+IPAG-12</Font>
	<Font>Noto Sans CJK JP-12</Font>

The font-controlling files are found:

Code: Select all

/etc/xdg/templates/_root_.jwmrc
/root/.jwmrc
/root/.jwm/jwmrc-theme
/root/.jwm/themes/*-jwmrc

This is not a perfect solution, as it seems font aliases do not work for nonlatin characters in jwm, see replies to quoted post for more details.

[MochiMoppel]

So I'm cleaning out my bookmarks and went to view a Japanese website

I guess it's https://tsujidotimes.com/

none of the characters displayed! Hello?? Why????

Hello!!
You need to have a font installed that supports Japanese characters. Do you? The site looks fine to me in Pale Moon.

I have no idea what this says, it's just a string of boxes!

Google Translate could help:

Let's enjoy old Japan at Tsujido
Tsujido is located in Fujisawa City, adjacent to Kamakura City, which is extremely popular with foreign tourists. There are no particularly prominent tourist destinations here, but it is easy to access some of Kanagawa Prefecture's most famous tourist destinations, such as Kamakura and Enoshima. In addition, it is a region called "Shonan", and it is a place close to the coast. Fujisawa City is also an area that prospered as a post town on the Tokaido in the Edo period, and the elegant townscape is still left.

Nice that they try to promote Kamakura . One of the best places to live in Japan ... but I'm biased

[orineu]

You need to have a font installed that supports Japanese characters. Do you? The site looks fine to me in Pale Moon.

Glad it works for you. Can you help me with figuring out how to install a system font that supports utf-8?

[MochiMoppel]

Can you help me with figuring out how to install a system font that supports utf-8?

https://oldforum.puppylinux.com/viewtop ... 16#p733116

[orineu]

Can you help me with figuring out how to install a system font that supports utf-8?

https://oldforum.puppylinux.com/viewtop ... 16#p733116

Am I supposed to install all of these pets, or the sfs? Can I not just drag and drop a ttf into a fonts folder? Do I have to delete the bad ttf?

[MochiMoppel]

Am I supposed to install all of these pets, or the sfs? Can I not just drag and drop a ttf into a fonts folder?

Yes you can.
Just try what the bloke in the 3rd post suggested.

[Phoenix]

There is also simply installing noto-cjk from the ppm, and then running

Code: Select all

fontcache -f

[orineu]

Am I supposed to install all of these pets, or the sfs? Can I not just drag and drop a ttf into a fonts folder?

Yes you can.
Just try what the bloke in the 3rd post suggested.

I'm assuming you're referring to your own post from 2013? That ftp requires a username and password.

[orineu]

There is also simply installing noto-cjk from the ppm, and then running

Code: Select all

fontcache -f

Thank you! This advice was helpful.

I wound up installing both fonts-noto-cjk-extra_20190410+repack1-2 (looked most complete, apparently included fonts-noto-cjk_20190410+repack1-2) and fonts-noto-ui-extra_20200323. Unfortunately, in exactly the same manner as the sfs i got from this post for Puppy 4.x , the "x" window title still does not display correctly (see attachments - same behavior in all applications such as Viewnior and, most importantly, the window-switcher bar). I would like to direct special attention to the attachment of ROX-Filer behavior - although the text displays correctly as a filename, the title bar becomes nonfunctional.

edit: this behavior is persistant after removing fonts-noto-ui-extra_20200323.

also, fontcache wasn't recognized as a command in terminal, but the fonts changed after a restart so that's no major issue.

[williams2]

In each case, the filename in the windows titlebar is not displayed correctly.

If you are using the JWM window manager, the titlebars are created by JWM.
In any case, it is the window manager that you are using that is creating the titlebars.

Maybe you can use JWMdesk to configure the titlebar font.

The titlebar font is configured in the hidden file /root/.jwmrc
There are other .jwm files in /root/
and there are more jwm configuration files in /root/.jwm/
Each JWM theme can set it's own font.

Which of these JWM configuration files is setting your titlebar font, I do not know.

You could try editing /root/.jwmrc.
The titlebar font in .jwmrc is set from these lines:

Code: Select all

<WindowStyle>
	<Font>DejaVu Sans-10</Font>

Maybe if you deleted the line <Font>DejaVu Sans-10</Font> it would work?
Or it might need to be set explicitly.

Then you need to run jwm -restart

if editing /root/.jwmrc works, you would need to make the same changes in /etc/xdg/templates/_root_.jwmrc

[MochiMoppel]

I'm assuming you're referring to your own post from 2013? That ftp requires a username and password.

The link still works for me. No username or password.
ftp://210.159.71.23/pub/linux/puppylinu ... P+IPAG.ttf
It's a Japanese IP Address. Maybe it blocks access from your location? No idea.

[greengeek]

When i try to access it i get a message saying "unknown url scheme" ( but i am away from PC and using Android at the moment). @Orineu - how are you accessing the site and can you post the error message?

[orineu]

When i try to access it i get a message saying "unknown url scheme" ( but i am away from PC and using Android at the moment). @Orineu - how are you accessing the site and can you post the error message?

It's an FTP url, so it opens in the ftp client, and it asks me for a username and password. If i click cancel, turns out i can access the ftp server anonymously, but i didn't try that before. also it takes you to the top level of the server so that was annoying.

[orineu]

In each case, the filename in the windows titlebar is not displayed correctly.

If you are using the JWM window manager, the titlebars are created by JWM.
In any case, it is the window manager that you are using that is creating the titlebars.

Maybe you can use JWMdesk to configure the titlebar font.

The titlebar font is configured in the hidden file /root/.jwmrc
There are other .jwm files in /root/
and there are more jwm configuration files in /root/.jwm/
Each JWM theme can set it's own font.

Which of these JWM configuration files is setting your titlebar font, I do not know.

You could try editing /root/.jwmrc.
The titlebar font in .jwmrc is set from these lines:

Code: Select all

<WindowStyle>
	<Font>DejaVu Sans-10</Font>

Maybe if you deleted the line <Font>DejaVu Sans-10</Font> it would work?
Or it might need to be set explicitly.

Then you need to run jwm -restart

if editing /root/.jwmrc works, you would need to make the same changes in /etc/xdg/templates/_root_.jwmrc

Oh, i got so excited! This definitely looked like it's related to the problem, this is what's in my jwmrc:

Code: Select all

<WindowStyle>
<Font>DejaVu Sans-10</Font>
<Width>6</Width>
<Height>29</Height>
<Foreground>#aaaaaa</Foreground>
<Background>#fffa71:#ffda51</Background>
<Corner>0</Corner>
<Outline>gray70</Outline>
<Active>
<Foreground>black</Foreground>
<Background>#ffda51:#ffa100</Background>
<Outline>black</Outline>
</Active>
</WindowStyle>

But, in the "installed fonts" display window, all fonts display the japanese characters just fine, even DejaVu Sans;

And more pertinently no matter what font name I put in that line, or if I deleted it entirely, the text in the title bars still displays only as boxes not as characters.

Next, I changed all the font names in the /.jwm/jwmrc-theme file to Noto Sans, did jwm -restart, but still, no change. By adding the -10 (or -15, etc) I can adjust the size of the font, so this does seem to be controlling something, but why is it not changing the font? Perhaps I have the name wrong.

In the JWM manual I found:

WINDOW STYLE

The WindowStyle tag controls the look of window borders. This tag supports the following attribute: decorations
The window decorations to use. Valid options are flat and motif. flat is the default.

Within this tag, the following tags are supported: Font
The font used for title bars. See the FONTS section for more information. This tag supports the following attribute: align
The window title alignment. Valid options are left, right, and center. left is the default.

and

FONTS

Fonts for various parts of JWM are specified within a Font tag. The text of this tag determines the font to use. This can be either a standard X font string or, if compiled with XFT support, an XFT font string.

The joe wing website is a bit more useful, https://joewing.net/projects/jwm/fonts.html

JWM supports core X11 fonts and, if compiled with Xft support, Xft fonts.

Core X11 Fonts:

Fonts begin with "-" and have "-" between each section. "*" may be used as a wildcard to match zero or more characters and "?" may be used to match any single character. It is generally recommended that the following fields be specified to get the desired font: family, weight, slant, and point size.

The format for an Xft font name is:

family-size:name=value...

Here family represents the font family ("times", "courier", etc.). size is the font size in points. Finally, additional font attributes may follow the ":". The following attributes are supported:

Somehow I got it! Noto and the other fonts I tried must not be "X11 fonts", whatever that means. But the M+1P+IPAG font worked, I just had to directly specify it in the jwmrc-theme file - installing it alone did not work.

Code: Select all

<WindowStyle>
	<Font>M+1P+IPAG-18</Font>
	<Width>3</Width>    
	<Foreground>#777777</Foreground
	<Background>#444444</Background>
	<Outline>black</Outline>
	<Active><Foreground>white</Foreground>
	<Background>#444444</Background>
	<Outline>black</Outline></Active>
</WindowStyle>

Last edit: I give up.... messing around with this tutorial (https://twiserandom.com/unix/x11-fonts- ... index.html) i think I figured out how to get the X11 coded name of the fonts, i think I figured out how to get the Noto font folder added to path using xset, but, no matter how I put in aliases or X11 font names, I can't get anything but the M+1P+IPAG font to change the font face displayed. Migu and MigMix, which are descendent fonts (http://mix-mplus-ipa.osdn.jp/migmix/), i can't get those to work either. Wildcard (*) doesn't work in the jwmrc-theme file either.

[thinkpadfreak]

And more pertinently no matter what font name I put in that line, or if I deleted it entirely, the text in the title bars still displays only as boxes not as characters.

Hello.
Some of the things we do when we create a Puppy Linux Japanese edition is as follows.

In the files
/etc/xdg/templates/_root_.jwmrc
/root/.jwm/jwmrc-theme
/root/.jwm/themes/*-jwmrc
"DejaVu Sans" should all be changed to "Sans."

For example,

Code: Select all

<WindowStyle>
    <Font>Sans-10</Font>

Then execute the command
# fixmenus
and then restart jwm.

Every time a new menu item is added, /root/.jwmrc is re-written referring to the template _root_.jwmrc. So, first of all, the font settings in _root_.jwmrc should be modified.

Somehow jwm does not deal with Japanese well when "DejaVu Sans" is designated.

[orineu]

"DejaVu Sans" should all be changed to "Sans."

...

Somehow jwm does not deal with Japanese well when "DejaVu Sans" is designated.

Hello!

Unfortunately "Sans" and "Dejavu Sans" have no difference on my machine, perhaps because Deja Vu is the first Sans Serif font the font selector happens upon? Regardless, I have the same display issue with using Monospace (although it displays a monospace font) and Serif (which displays a serif font). It seems like the default fonts installed do not support utf-8 and I don't know how to remove/replace/update them.

[orineu]

UPDATE: Fixed

So, now I can specify which font to use, by the name that shows up in gfontsel, and it actually shows up. It must have required a full restart, like perhaps a font list somewhere needed to be rebuilt, and that was automatically done at startup? I noticed this because I had (rather sloppily) left a cursive font defined for the menus in the jwmrc theme file, and when I restarted my PC it had been applied; now, however, all I need to do is #jwm -restart to see previously installed fonts displayed correctly.

So to recap, I think this is what wound up working:

1. install a font using the JWMdesk Font Manager (Migu, MigMix, or M+, for instance) or PPM (Noto-cjk, for instance)
2. restart the entire system (not just the graphical server/jwm)
3. In the font controlling files, specify the exact name of a font (as displayed in gfontsel, may need to resize the window) that supports your charset (do not rely on gfontsel to determine this) followed by the size as below:

Code: Select all

	<Font>Migu 1P-12</Font>
	<Font>MigMix 1P-12</Font>
	<Font>M+1P+IPAG-12</Font>
	<Font>Noto Sans CJK JP-12</Font>

Again, the font-controlling files are found:

/etc/xdg/templates/_root_.jwmrc
/root/.jwmrc
/root/.jwm/jwmrc-theme
/root/.jwm/themes/*-jwmrc

Further questions:
1. I wonder if this entire experience can be confirmed by someone else?
2. Why did (only) M+1P+IPAG work before a system restart?
3. Is it required to manually add folders to the font path using xset, or rebuild the font paths? Or can you avoid the restart by updating something I didn't change, such as "font cache"? (how would that be done??)
4. How do you check what characters a specific font *actually* supports? gfontsel appears to "fill in" missing characters somehow, but that was misleading behavior.
5. Is there any way to specify a font hierarchy, for instance a cursive font for latin characters as well as one of the CJK or IPA fonts for extended characters, side-by-side?

[thinkpadfreak]

Further questions:
1. I wonder if this entire experience can be confirmed by someone else?
2. Why did (only) M+1P+IPAG work before a system restart?
3. Is it required to manually add folders to the font path using xset, or rebuild the font paths? Or can you avoid the restart by updating something I didn't change, such as "font cache"? (how would that be done??)
4. How do you check what characters a specific font *actually* supports? gfontsel appears to "fill in" missing characters somehow, but that was misleading behavior.
5. Is there any way to specify a font hierarchy, for instance a cursive font for latin characters as well as one of the CJK or IPA fonts for extended characters, side-by-side?

I cannot answer all the questions, but here are a few things I do.

To update the font cache, I execute
# fc-cache -fv
but it seems this is not necessary if the computer is rebooted.

As far as I know, /etc/fonts/conf.avail/65-nonlatin.conf is used to control the preference of nonlatin fonts including Japanese fonts. (The location of the file may be different depending on the distribution.)

But the above configuration is overridden by /etc/fonts/local.conf.
The following is an example of local.conf:

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<!-- /etc/fonts/local.conf file for local customizations -->
<fontconfig>

    <match target="pattern">
        <test qual="any" name="family">
            <string>serif</string>
        </test>
        <edit name="family" mode="append" binding="same">
            <string>TakaoExMincho</string>
        </edit>
    </match>

</fontconfig>

In this local.conf, I designate TakaoExMincho as "serif." In the same way you can designate a font as "Sans." The font will be automatically used as "Sans" by applications.

[orineu]

As far as I know, /etc/fonts/conf.avail/65-nonlatin.conf is used to control the preference of nonlatin fonts including Japanese fonts. (The location of the file may be different depending on the distribution.)

But the above configuration is overridden by /etc/fonts/local.conf.

Aha, on my installation, there are no fonts specified in local.conf, and in 65-nonlatin.conf there are many fonts specified, but not a single one of them are actually installed. So, I've created a "cursive" alias, added MigMix 1P as the preferred font in the 65-nonlatin.conf and added Sketch to the preexisting cursive alias in the 60-latin.conf file, and changed the jwmrc-theme to use a cursive font. Updating the font cache and restarting the jwm changed the font back to the broken default font, so I'm going to try a restart.

After a restart, the cursive font is displaying for the latin characters, but MigMix 1P is not displaying for the nonlatin characters. So I found one of the ja fonts named in the file, "AR PL ShanHeiSun Uni" downloaded and installed it, but a font cache update and jwm restart did nothing (as expected because we're still in cursive at this point); so I changed the jwmrc-theme to serif and observed the title bar latin characters change to serif; but the japanese characters still fail to display in the jwm. Performed a restart...

Nothing has changed.

Have confirmed that the aliases that are accepted by the jwmrc-theme file are "serif", "sans" (but not "sans-serif", which is the alias used in the 60-latin.conf file), "monospace", and now (because I added a supported font to the conf,) "cursive".

For the science of it, I grabbed the name of the nonlatin font that I downloaded and put it into the 60-latin.conf file, see you after the restart...

No change.

Added "known good" font M+1P+IPAG to the top of the serif fonts in 60-latin.conf. Reboot. No change.

Conclusion: nonlatin characters do not display correctly when font is controlled by an alias, regardless of whether a supporting font is correctly defined in the alias configuration files 60-latin.conf and/or 65-nonlatin.conf.

Removed all changes I made to 60-latin.conf and 65-nonlatin.conf. Naming "AR PL ShanHeiSun Uni" exactly in the jwmrc-theme file displays the correct font.

(edit: i made a mistake: the sans-serif japanese supporting font in the attachment, which i deleted, is actually not MigMix 1P, I'm uncertain of what it is at the moment haha but the way "mo" is drawn is different)

[thinkpadfreak]

Conclusion: nonlatin characters do not display correctly when font is controlled by an alias

There may be some other elements which affect the font configuration.

I installed "MigMix 1P" onto a Japanized Bionicpup64 (which I arranged myself). Designating the font as "sans" was not successful. But by designating it as "sans-serif," jwm and applications accepted the setting.

Bionicpup64 Japanized edition
https://sakurapup.com/forum1/viewtopic.php?f=24&t=3450

local.conf

Code: Select all

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<!-- /etc/fonts/local.conf file for local customizations -->
<fontconfig>
<!--  Enable sub-pixel rendering -->
	<match target="font">
		<test qual="all" name="rgba">
			<const>unknown</const>
		</test>
		<edit name="rgba" mode="assign"><const>rgb</const></edit>
	</match>

<!-- suggested by 'upnorth' http://murga-linux.com/puppy/viewtopic.php?t=54294&start=15
 Reject bitmap fonts -->
 <selectfont>
  <rejectfont>
   <pattern>
     <patelt name="scalable"><bool>false</bool></patelt>
   </pattern>
  </rejectfont>
 </selectfont>

    <match target="pattern">
        <test qual="any" name="family">
            <string>sans-serif</string>
        </test>
        <edit name="family" mode="append" binding="same">
            <string>MigMix 1P</string>
        </edit>
    </match>

	<match target="pattern">
        <test qual="any" name="family">
            <string>serif</string>
        </test>
        <edit name="family" mode="append" binding="same">
            <string>TakaoExMincho</string>
        </edit>
    </match>

	<!-- hinting -->
	<match target="font">
		<edit name="autohint" mode="assign">
			<bool>true</bool>
		</edit>
	</match>

</fontconfig>

The font settings of jwm are all "Sans-10."

In the attached picture, jwm and Chrome display the 8th tuplet, which is characteristic of MigMix 1P. M+1P+IPAG does not include the character.

(Somehow the picture is shrunk when it is uploaded. Opening the picture as a new tab will enlarge it.)

[orineu]

There may be some other elements which affect the font configuration.

I installed "MigMix 1P" onto a Japanized Bionicpup64 (which I arranged myself). Designating the font as "sans" was not successful. But by designating it as "sans-serif," jwm and applications accepted the setting.

Come to think of it, my local.conf doesn't contain any font alias specifiers, so I didn't mess with it, thinking that it was "simply an override". So, I add MigMix 1P (a sans-serif font) to each of the font categories (even though this is incorrect) to match your settings... restarted, no change. Using the "Serif" alias in jwmrc-theme still uses Deja Vu.

Gfontsel displays a separate font for each major family, named "Sans", "Serif", and "Monospace" that is a clone of DejaVu Sans, DejaVu Serif, and DejaVu Sans Mono respectively. I wonder if the presence of these fonts are causing issues?

Incidentally, accidentally found this bit of knowledge: Sounds like "sans" should be accepted as an alias according to this bit located in fonts.conf after line 61, although "replacing with" does make it seem like maybe "accept alternate" is incorrect language, although it is interesting that jwm accepts both "Sans" and "Sans Serif" but not "sans-serif"

Code: Select all

<!--
  Accept deprecated 'sans' alias, replacing it with 'sans-serif'
-->
	<match target="pattern">
		<test qual="any" name="family">
			<string>sans</string>
		</test>
		<edit name="family" mode="assign" binding="same">
			<string>sans-serif</string>
		</edit>
	</match>

Here's another accidental finding that might be related... in /etc/fonts/conf.avail/57-dejavu-sans.conf I found this bit where Deja Vu is set as the default sans-serif font, presumably each of the *-dejavu-*.conf files has something similar....

Code: Select all

  <!-- Generic name assignment -->
  <alias>
    <family>DejaVu Sans</family>
    <default>
      <family>sans-serif</family>
    </default>
  </alias>
  <!-- Generic name aliasing -->
  <alias>
    <family>sans-serif</family>
    <prefer>
      <family>DejaVu Sans</family>
    </prefer>
  </alias>

One last incidental observation: Default fonts inside the windows of apps are also inconsistent, and I can't find that these are configurable on a application level:

Rox-filer uses M+1P+IPAG
Chrome uses Noto Sans CJK JP unless website specifies a compatible font
Geany (file titles) also uses Noto Sans CJK JP

EDIT: Ultimately I would like to use something like Garamond AND Hiragino Mincho for serif; Futura or Graphik AND MigMix 1P for sans-serif; Sketch AND Mogihapen for cursive...